05_finetune/12_fine_tune_squad_falcon_sagemaker_jumpstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9fb44f6d",
   "metadata": {},
   "source": [
    "# Introduction to SageMaker JumpStart - Text Generation with Falcon models\n",
    "\n",
    "---\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy and fine-tuning Falcon models for text generation. For inference, we show several example use cases including code generation, question answering, translation etc. For fine-tuning, we include two types of fine-tuning: instruction fine-tuning and domain adaption fine-tuning. \n",
    "\n",
    "The Falcon model is a permissively licensed ([Apache-2.0](https://jumpstart-cache-prod-us-east-2.s3.us-east-2.amazonaws.com/licenses/Apache-License/LICENSE-2.0.txt)) open source model trained on the [RefinedWeb dataset](https://huggingface.co/datasets/tiiuae/falcon-refinedweb). \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5c76151-390e-4592-a192-a857292e4e53",
   "metadata": {},
   "source": [
    "Below is the content of the notebook.\n",
    "\n",
    "1. [Deploy Falcon model for inference](#1.-Deploying-Falcon-model-for-inference)\n",
    "   * [1.1. Changing instance type](#1.1.-Changing-instance-type)\n",
    "   * [1.2. Changing number of GPUs](#1.2.-Changing-number-of-GPUs)\n",
    "   * [1.3. About the model](#1.3.-About-the-model)\n",
    "   * [1.4. Supported parameters](#1.4.-Supported-parameters)\n",
    "2. [Instruction fine-tuning](#2.-Instruction-fine-tuning)\n",
    "   * [2.1. Preparing training data](#2.1.-Preparing-training-data)\n",
    "   * [2.2. Prepare training parameters](#2.2.-Prepare-training-parameters)\n",
    "   * [2.3. Starting training](#2.3.-Starting-training)\n",
    "   * [2.4. Deploying inference endpoints](#2.4.-Deploying-inference-endpoints)\n",
    "   * [2.5. Running inference queries and compare model performances](#2.5.-Running-inference-queries-and-compare-model-performances)\n",
    "   * [2.6. Clean up endpoint](#2.6.-Clean-up-the-endpoint)\n",
    "3. [Domain adaptation fine-tuning](#3.-Domain-adaptation-fine-tuning)\n",
    "   * [3.1. Preparing training data](#3.1.-Preparing-training-data)\n",
    "   * [3.2. Prepare training parameters](#3.2.-Prepare-training-parameters)\n",
    "   * [3.3. Starting training](#3.3.-Starting-training)\n",
    "   * [3.4. Deploying inference endpoints](#3.4.-Deploying-inference-endpoints)\n",
    "   * [3.5. Running inference queries and compare model performances](#3.5.-Running-inference-queries-and-compare-model-performances)\n",
    "   * [3.6. Clean up endpoint](#3.6.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ee26a6-7979-4a12-8f43-0089728bf869",
   "metadata": {},
   "source": [
    "## 1. Deploying Falcon model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9b05b931-992e-4526-978d-f03196874a3b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "daal4py 2021.6.0 requires daal==2021.4.0, which is not installed.\n",
      "spyder 5.3.3 requires pyqt5<5.16, which is not installed.\n",
      "spyder 5.3.3 requires pyqtwebengine<5.16, which is not installed.\n",
      "autovizwidget 0.21.0 requires pandas<2.0.0,>=0.20.1, but you have pandas 2.1.1 which is incompatible.\n",
      "awscli 1.29.63 requires botocore==1.31.63, but you have botocore 1.31.65 which is incompatible.\n",
      "distributed 2022.7.0 requires tornado<6.2,>=6.0.3, but you have tornado 6.3.3 which is incompatible.\n",
      "hdijupyterutils 0.21.0 requires pandas<2.0.0,>=0.17.1, but you have pandas 2.1.1 which is incompatible.\n",
      "jupyterlab 3.4.4 requires jupyter-server~=1.16, but you have jupyter-server 2.7.3 which is incompatible.\n",
      "jupyterlab-server 2.10.3 requires jupyter-server~=1.4, but you have jupyter-server 2.7.3 which is incompatible.\n",
      "numba 0.55.1 requires numpy<1.22,>=1.18, but you have numpy 1.26.1 which is incompatible.\n",
      "panel 0.13.1 requires bokeh<2.5.0,>=2.4.0, but you have bokeh 3.3.0 which is incompatible.\n",
      "sagemaker-datawrangler 0.4.3 requires sagemaker-data-insights==0.4.0, but you have sagemaker-data-insights 0.3.3 which is incompatible.\n",
      "spyder 5.3.3 requires ipython<8.0.0,>=7.31.1, but you have ipython 8.16.1 which is incompatible.\n",
      "spyder 5.3.3 requires pylint<3.0,>=2.5.0, but you have pylint 3.0.1 which is incompatible.\n",
      "spyder-kernels 2.3.3 requires ipython<8,>=7.31.1; python_version >= \"3\", but you have ipython 8.16.1 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install sagemaker --quiet --upgrade --force-reinstall\n",
    "!pip install ipywidgets==7.0.0 --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "20a31047",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version = \"huggingface-llm-falcon-40b-instruct-bf16\", \"*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n",
      "--------------------------!CPU times: user 1.45 s, sys: 294 ms, total: 1.74 s\n",
      "Wall time: 13min 36s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "my_model = JumpStartModel(model_id=model_id)\n",
    "predictor = my_model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67abf8ea-16c7-4d55-8500-bfd2a16d1294",
   "metadata": {},
   "source": [
    "### 1.1. Changing instance type\n",
    "---\n",
    "\n",
    "\n",
    "Models have been tested on the following instance types:\n",
    "\n",
    " - Falcon 7B and 7B instruct: `ml.g5.2xlarge`, `ml.g5.2xlarge`, `ml.g5.4xlarge`, `ml.g5.8xlarge`, `ml.g5.16xlarge`, `ml.g5.12xlarge`, `ml.g5.24xlarge`, `ml.g5.48xlarge`, `ml.p4d.24xlarge`\n",
    " - Falcon 40B and 40B instruct: `ml.g5.12xlarge`, `ml.g5.48xlarge`, `ml.p4d.24xlarge`\n",
    "\n",
    "If an instance type is not available in you region, please try a different instance. You can do so by specifying instance type in the JumpStartModel class.\n",
    "\n",
    "`my_model = JumpStartModel(model_id=\"huggingface-llm-falcon-40b-instruct-bf16\", instance_type=\"ml.g5.12xlarge\")`\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23b42484-1770-4084-887c-c48ffccc02cc",
   "metadata": {},
   "source": [
    "### 1.2. Changing number of GPUs\n",
    "---\n",
    "Falcon models are served with HuggingFace (HF) LLM DLC which requires specifying number of GPUs during model deployment. \n",
    "\n",
    "**Falcon 7B and 7B instruct:** HF LLM DLC currently does not support sharding for 7B model. Thus, even if more than one GPU is available on the instance, please do not increase number of GPUs. \n",
    "\n",
    "**Falcon 40B and 40B instruct:** By default number of GPUs are set to 4. However, if you are using `ml.g5.48xlarge` or `ml.p4d.24xlarge`, you can increase number of GPUs to be 8 as follows: \n",
    "\n",
    "`my_model = JumpStartModel(model_id=\"huggingface-llm-falcon-40b-instruct-bf16\", instance_type=\"ml.g5.48xlarge\")`\n",
    "\n",
    "`my_model.env['SM_NUM_GPUS'] = '8'`\n",
    "\n",
    "`predictor = my_model.deploy()`\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c1a55aa5-f2ad-4db8-9718-b76f969cffbe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Amazon SageMaker is a fully managed platform that enables data scientists and developers to quickly and easily build, train, and deploy machine learning models at scale. It includes a broad set of capabilities including pre-built machine learning algorithms, fully-managed training, inference, and deployment services, and a marketplace for pre-trained models. SageMaker also supports popular machine learning frameworks and programming languages, and integrates with other AWS services to make it easier to build end-to-end machine learning solutions.\n",
      "CPU times: user 12.2 ms, sys: 7.23 ms, total: 19.5 ms\n",
      "Wall time: 5.21 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "prompt = \"Tell me about Amazon SageMaker.\"\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.9,\n",
    "        \"temperature\": 0.8,\n",
    "        \"max_new_tokens\": 1024,\n",
    "        \"stop\": [\"<|endoftext|>\", \"</s>\"]\n",
    "    }\n",
    "}\n",
    "\n",
    "response = predictor.predict(payload)\n",
    "print(response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15da465e-b855-4249-93b8-f80a3627a62b",
   "metadata": {
    "jumpStartAlterations": [],
    "tags": []
   },
   "source": [
    "### 1.3. About the model\n",
    "\n",
    "---\n",
    "Falcon is a causal decoder-only model built by [Technology Innovation Institute](https://www.tii.ae/) (TII) and trained on more than 1 trillion tokens of RefinedWeb enhanced with curated corpora. It was built using custom-built tooling for data pre-processing and model training built on Amazon SageMaker. As of June 6, 2023, it is the best open-source model currently available. Falcon-40B outperforms LLaMA, StableLM, RedPajama, MPT, etc. To see comparison, see [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). It features an architecture optimized for inference, with FlashAttention and multiquery. \n",
    "\n",
    "\n",
    "[Refined Web Dataset](https://huggingface.co/datasets/tiiuae/falcon-refinedweb): Falcon RefinedWeb is a massive English web dataset built by TII and released under an Apache 2.0 license. It is a highly filtered dataset with large scale de-duplication of CommonCrawl. It is observed that models trained on RefinedWeb achieve performance equal to or better than performance achieved by training model on curated datasets, while only relying on web data.\n",
    "\n",
    "**Model Sizes:**\n",
    "- **Falcon-7b**: It is a 7 billion parameter model trained on 1.5 trillion tokens. It outperforms comparable open-source models (e.g., MPT-7B, StableLM, RedPajama etc.). To see comparison, see [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). To use this model, please select `model_id` in the cell above to be \"huggingface-llm-falcon-7b-bf16\".\n",
    "- **Falcon-40B**: It is a 40 billion parameter model trained on 1 trillion tokens.  It has surpassed renowned models like LLaMA-65B, StableLM, RedPajama and MPT on the public leaderboard maintained by Hugging Face, demonstrating its exceptional performance without specialized fine-tuning. To see comparison, see [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). \n",
    "\n",
    "**Instruct models (Falcon-7b-instruct/Falcon-40B-instruct):** Instruct models are base falcon models fine-tuned on a mixture of chat and instruction datasets. They are ready-to-use chat/instruct models.  To use these models, please select `model_id` in the cell above to be \"huggingface-textgeneration-falcon-7b-instruct-bf16\" or \"huggingface-textgeneration-falcon-40b-instruct-bf16\".\n",
    "\n",
    "It is [recommended](https://huggingface.co/tiiuae/falcon-7b) that Instruct models should be used without fine-tuning and base models should be fine-tuned further on the specific task.\n",
    "\n",
    "**Limitations:**\n",
    "\n",
    "- Falcon models are mostly trained on English data and may not generalize to other languages. \n",
    "- Falcon carries the stereotypes and biases commonly encountered online and in the training data. Hence, it is recommended to develop guardrails and to take appropriate precautions for any production use. This is a raw, pretrained model, which should be further finetuned for most usecases.\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "719560fd-c9b2-4d4c-a7de-914b8aa72557",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def query_endpoint(payload):\n",
    "    \"\"\"Query endpoint and print the response\"\"\"\n",
    "    response = predictor.predict(payload)\n",
    "    print(f\"\\033[1m Input:\\033[0m {payload['inputs']}\")\n",
    "    print(f\"\\033[1m Output:\\033[0m {response[0]['generated_text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "009490f0-fb8a-4d92-b2ad-85e53223922a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m Write a program to compute factorial in python:\n",
      "\u001b[1m Output:\u001b[0m \n",
      "You can compute factorial in Python using the built-in function `math.factorial()`. Here's an example:\n",
      "\n",
      "```python\n",
      "import math\n",
      "\n",
      "n = 5\n",
      "factorial = math.factorial(n)\n",
      "print(factorial)\n",
      "```\n",
      "\n",
      "This will output `120`, which is the factorial of 5.\n"
     ]
    }
   ],
   "source": [
    "# Code generation\n",
    "payload = {\"inputs\": \"Write a program to compute factorial in python:\", \"parameters\":{\"max_new_tokens\": 200}}\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7816dd22-fd8f-4374-ba9d-a62b941ebb16",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m Building a website can be done in 10 simple steps:\n",
      "\u001b[1m Output:\u001b[0m \n",
      "\n",
      "1. Choose a domain name and hosting provider.\n",
      "2. Install a content management system (CMS) like WordPress.\n",
      "3. Choose a theme and customize it to your liking.\n",
      "4. Create pages and add content.\n",
      "5. Optimize your website for search engines.\n",
      "6. Test your website for usability and functionality.\n",
      "7. Launch your website and promote it.\n",
      "8. Monitor your website's performance and make improvements.\n",
      "9. Maintain your website by updating it regularly.\n",
      "10. Continuously improve\n"
     ]
    }
   ],
   "source": [
    "payload = {\n",
    "    \"inputs\": \"Building a website can be done in 10 simple steps:\",\n",
    "    \"parameters\":{\n",
    "        \"max_new_tokens\": 110,\n",
    "        \"no_repeat_ngram_size\": 3\n",
    "        }\n",
    "}\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c4aac6cc-d282-428a-b334-5dcdd505e480",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m Translate English to French:\n",
      "\n",
      "    sea otter => loutre de mer\n",
      "\n",
      "    peppermint => menthe poivrée\n",
      "\n",
      "    plush girafe => girafe peluche\n",
      "\n",
      "    cheese =>\n",
      "\u001b[1m Output:\u001b[0m  fromage\n",
      "\n",
      "   \n"
     ]
    }
   ],
   "source": [
    "# Translation\n",
    "payload = {\n",
    "    \"inputs\": \"\"\"Translate English to French:\n",
    "\n",
    "    sea otter => loutre de mer\n",
    "\n",
    "    peppermint => menthe poivrée\n",
    "\n",
    "    plush girafe => girafe peluche\n",
    "\n",
    "    cheese =>\"\"\",\n",
    "    \"parameters\":{\n",
    "        \"max_new_tokens\": 3\n",
    "    }\n",
    "}\n",
    "\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "117b6645-56c5-4f0d-bbab-75bae8955943",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m \"I hate it when my phone battery dies.\"\n",
      "                Sentiment: Negative\n",
      "                ###\n",
      "                Tweet: \"My day has been :+1:\"\n",
      "                Sentiment: Positive\n",
      "                ###\n",
      "                Tweet: \"This is the link to the article\"\n",
      "                Sentiment: Neutral\n",
      "                ###\n",
      "                Tweet: \"This new music video was incredibile\"\n",
      "                Sentiment:\n",
      "\u001b[1m Output:\u001b[0m  Positive\n",
      "                \n"
     ]
    }
   ],
   "source": [
    "# Sentiment-analysis\n",
    "payload = {\n",
    "    \"inputs\": \"\"\"\"I hate it when my phone battery dies.\"\n",
    "                Sentiment: Negative\n",
    "                ###\n",
    "                Tweet: \"My day has been :+1:\"\n",
    "                Sentiment: Positive\n",
    "                ###\n",
    "                Tweet: \"This is the link to the article\"\n",
    "                Sentiment: Neutral\n",
    "                ###\n",
    "                Tweet: \"This new music video was incredibile\"\n",
    "                Sentiment:\"\"\",\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\":2\n",
    "    }\n",
    "}\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4594ede3-1272-4e56-8926-ccaf9e66f314",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m Could you remind me when was the C programming language invented?\n",
      "\u001b[1m Output:\u001b[0m \n",
      "The C programming language was invented in the early 1970s by Dennis Ritchie at Bell Labs.\n"
     ]
    }
   ],
   "source": [
    "# Question answering\n",
    "payload = {\n",
    "    \"inputs\": \"Could you remind me when was the C programming language invented?\",\n",
    "    \"parameters\":{\n",
    "        \"max_new_tokens\": 50\n",
    "    }\n",
    "}\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fa8d82be-3c33-47c8-b0d7-d8675484b1d7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m What is the recipe for a delicious lemon cheesecake?\n",
      "\u001b[1m Output:\u001b[0m \n",
      "Ingredients:\n",
      "- 1 1/2 cups graham cracker crumbs\n",
      "- 1/4 cup granulated sugar\n",
      "- 1/3 cup unsalted butter, melted\n",
      "- 3 (8-ounce) packages cream cheese, softened\n",
      "- 1 1/2 cups granulated sugar\n",
      "- 1/4 cup all-purpose flour\n",
      "- 1/2 cup sour cream\n",
      "- 1/2 cup heavy cream\n",
      "- 1 tablespoon lemon zest\n",
      "- 1/4 cup fresh lemon juice\n",
      "- 4 large eggs\n",
      "\n",
      "Instructions:\n",
      "1. Preheat oven to 325°F.\n",
      "2. In a medium bowl, combine graham cracker crumbs, sugar, and melted butter. Press mixture into the bottom of a 9-inch springform pan.\n",
      "3. In a large mixing bowl, beat cream cheese until smooth. Add sugar and flour, and beat until well combined.\n",
      "4. Add sour cream, heavy cream, lemon zest, and lemon juice, and beat until smooth.\n",
      "5. Add eggs one at a time, beating well after each addition.\n",
      "6. Pour mixture into prepared crust.\n",
      "7. Bake for 1 hour and 15 minutes, or until center is set.\n",
      "8. Allow cheesecake to cool completely before serving.\n"
     ]
    }
   ],
   "source": [
    "# Recipe generation\n",
    "payload = {\"inputs\": \"What is the recipe for a delicious lemon cheesecake?\", \"parameters\":{\"max_new_tokens\": 400}}\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d4f125c1-eae1-45ad-a065-651d9e13dfa8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Input:\u001b[0m Starting today, the state-of-the-art Falcon 40B foundation model from Technology\n",
      "    Innovation Institute (TII) is available on Amazon SageMaker JumpStart, SageMaker's machine learning (ML) hub\n",
      "    that offers pre-trained models, built-in algorithms, and pre-built solution templates to help you quickly get\n",
      "    started with ML. You can deploy and use this Falcon LLM with a few clicks in SageMaker Studio or\n",
      "    programmatically through the SageMaker Python SDK.\n",
      "    Falcon 40B is a 40-billion-parameter large language model (LLM) available under the Apache 2.0 license that\n",
      "    ranked #1 in Hugging Face Open LLM leaderboard, which tracks, ranks, and evaluates LLMs across multiple\n",
      "    benchmarks to identify top performing models. Since its release in May 2023, Falcon 40B has demonstrated\n",
      "    exceptional performance without specialized fine-tuning. To make it easier for customers to access this\n",
      "    state-of-the-art model, AWS has made Falcon 40B available to customers via Amazon SageMaker JumpStart.\n",
      "    Now customers can quickly and easily deploy their own Falcon 40B model and customize it to fit their specific\n",
      "    needs for applications such as translation, question answering, and summarizing information.\n",
      "    Falcon 40B are generally available today through Amazon SageMaker JumpStart in US East (Ohio),\n",
      "    US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Mumbai),\n",
      "    Europe (London), Europe (Frankfurt), Europe (Ireland), and Canada (Central),\n",
      "    with availability in additional AWS Regions coming soon. To learn how to use this new feature,\n",
      "    please see SageMaker JumpStart documentation, the Introduction to SageMaker JumpStart –\n",
      "    Text Generation with Falcon LLMs example notebook, and the blog Technology Innovation Institute trainsthe\n",
      "    state-of-the-art Falcon LLM 40B foundation model on Amazon SageMaker. Summarize the article above:\n",
      "\u001b[1m Output:\u001b[0m \n",
      "    Amazon SageMaker JumpStart now offers the state-of-the-art Falcon 40B foundation model from Technology\n",
      "    Innovation Institute (TII) as a pre-trained model. Customers can quickly and easily deploy their own Falcon 40B\n",
      "    model and customize it to fit their specific needs for applications such as translation, question answering,\n",
      "    and summarizing information. Falcon 40B is generally available today through Amazon SageMaker JumpStart in US East (Ohio),\n",
      "    US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Mumbai),\n",
      "    Europe (London), Europe (Frankfurt), Europe (Ireland), and Canada (Central),\n",
      "    with availability in additional AWS Regions coming soon.\n"
     ]
    }
   ],
   "source": [
    "# Summarization\n",
    "\n",
    "payload = {\n",
    "    \"inputs\":\"\"\"Starting today, the state-of-the-art Falcon 40B foundation model from Technology\n",
    "    Innovation Institute (TII) is available on Amazon SageMaker JumpStart, SageMaker's machine learning (ML) hub\n",
    "    that offers pre-trained models, built-in algorithms, and pre-built solution templates to help you quickly get\n",
    "    started with ML. You can deploy and use this Falcon LLM with a few clicks in SageMaker Studio or\n",
    "    programmatically through the SageMaker Python SDK.\n",
    "    Falcon 40B is a 40-billion-parameter large language model (LLM) available under the Apache 2.0 license that\n",
    "    ranked #1 in Hugging Face Open LLM leaderboard, which tracks, ranks, and evaluates LLMs across multiple\n",
    "    benchmarks to identify top performing models. Since its release in May 2023, Falcon 40B has demonstrated\n",
    "    exceptional performance without specialized fine-tuning. To make it easier for customers to access this\n",
    "    state-of-the-art model, AWS has made Falcon 40B available to customers via Amazon SageMaker JumpStart.\n",
    "    Now customers can quickly and easily deploy their own Falcon 40B model and customize it to fit their specific\n",
    "    needs for applications such as translation, question answering, and summarizing information.\n",
    "    Falcon 40B are generally available today through Amazon SageMaker JumpStart in US East (Ohio),\n",
    "    US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Mumbai),\n",
    "    Europe (London), Europe (Frankfurt), Europe (Ireland), and Canada (Central),\n",
    "    with availability in additional AWS Regions coming soon. To learn how to use this new feature,\n",
    "    please see SageMaker JumpStart documentation, the Introduction to SageMaker JumpStart –\n",
    "    Text Generation with Falcon LLMs example notebook, and the blog Technology Innovation Institute trainsthe\n",
    "    state-of-the-art Falcon LLM 40B foundation model on Amazon SageMaker. Summarize the article above:\"\"\",\n",
    "    \"parameters\":{\n",
    "        \"max_new_tokens\":200\n",
    "        }\n",
    "    }\n",
    "query_endpoint(payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "132d3fee-5ce2-4ff2-93fe-5779762ce2cf",
   "metadata": {},
   "source": [
    "### 1.4. Supported parameters\n",
    "\n",
    "***\n",
    "Some of the supported parameters while performing inference are the following:\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **max_new_tokens:** Model generates text until the output length (excluding the input context length) reaches `max_new_tokens`. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of sentence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelihood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **return_full_text:** If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.\n",
    "* **stop**: If specified, it must a list of strings. Text generation stops if any one of the specified strings is generated.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint. \n",
    "\n",
    "For more parameters and information on HF LLM DLC, please see [this article](https://huggingface.co/blog/sagemaker-huggingface-llm#4-run-inference-and-chat-with-our-model).\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3460e23d-7011-4639-94a2-c55f53a035c9",
   "metadata": {},
   "source": [
    "## 2. Instruction fine-tuning\n",
    "\n",
    "Now, we demonstrate how to instruction-tune `huggingface-llm-falcon-7b-instruct-bf16` model for a new task. As mentioned in [Section 1.3 About the models](#1.3.-About-the-model),  **Falcon-7b-instruct**/**Falcon-40B-instruct** models are instruction base falcon models fine-tuned on a mixture of chat and instruction datasets. \n",
    "\n",
    "**In this task, given a piece of context, the model is asked to generate questions that are relevant to the text, but `cannot` be answered based on provided information. Examples are given in the inference section of this notebook.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aba27120-e9ca-4408-b0f0-80ef5a1a5531",
   "metadata": {},
   "source": [
    "### 2.1. Preparing training data\n",
    "We will use a subset of SQuAD2.0 for supervised fine-tuning. This dataset contains questions posed by human annotators on a set of Wikipedia articles. In addition to questions with answers, SQuAD2.0 contains about 50k unanswerable questions. Such questions are plausible, but cannot be directly answered from the articles' content. We only use unanswerable questions for our task.\n",
    "\n",
    "Citation: @article{rajpurkar2018know, title={Know what you don't know: Unanswerable questions for SQuAD}, author={Rajpurkar, Pranav and Jia, Robin and Liang, Percy}, journal={arXiv preprint arXiv:1806.03822}, year={2018} }\n",
    "\n",
    "License: Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "976cb8a9-be2f-436b-9444-a52fd0caf518",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n",
      "\u001b[1maws_region:\u001b[0m us-west-2\n",
      "\u001b[1maws_role:\u001b[0m arn:aws:iam::079002598131:role/service-role/AmazonSageMaker-ExecutionRole-20220804T150518\n",
      "\u001b[1moutput_bucket:\u001b[0m sagemaker-us-west-2-079002598131\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import json\n",
    "\n",
    "# Get current region, role, and default bucket\n",
    "aws_region = boto3.Session().region_name\n",
    "aws_role = sagemaker.session.Session().get_caller_identity_arn()\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "\n",
    "# This will be useful for printing\n",
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "print(f\"{bold}aws_region:{unbold} {aws_region}\")\n",
    "print(f\"{bold}aws_role:{unbold} {aws_role}\")\n",
    "print(f\"{bold}output_bucket:{unbold} {output_bucket}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "01e61aa5-e2cd-4cec-902e-5a4b94bb5817",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['./train-v2.0.json']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "# We will use the train split of SQuAD2.0\n",
    "original_data_file = \"train-v2.0.json\"\n",
    "\n",
    "# The data was mirrored in the following bucket\n",
    "original_data_location = (\n",
    "    f\"s3://sagemaker-example-files-prod-{aws_region}/datasets/text/squad2.0/{original_data_file}\"\n",
    ")\n",
    "S3Downloader.download(original_data_location, \".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c061fc5-e11b-4a72-8726-00ab32452769",
   "metadata": {},
   "source": [
    "The training data must be formatted in JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample. All training data must be in a single folder, however it can be saved in multiple jsonl files. The .jsonl file extension is mandatory. The training folder can also contain a template.json file describing the input and output formats.\n",
    "\n",
    "If no template file is given, the following default template will be used:\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"prompt\": \"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}`,\n",
    "    \"completion\": \"{response}\",\n",
    "}\n",
    "```\n",
    "\n",
    "In this case, the data in the JSON lines entries must include `instruction`, `context`, and `response` fields.\n",
    "\n",
    "Different from using the default prompt template, in this demo we are going to use a custom template (see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "30df5174-667c-44ea-ad20-47c44ca0e628",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = {\n",
    "    \"prompt\": \"Ask a question which is related to the following text, but cannot be answered based on the text. Text: {context}\",\n",
    "    \"completion\": \"{question}\",\n",
    "}\n",
    "\n",
    "with open(\"template.json\", \"w\") as f:\n",
    "    json.dump(template, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cbdecf3-cb15-4ef9-ab6c-ebe9c43aab1d",
   "metadata": {},
   "source": [
    "Next, we are going to reformat the SQuAD 2.0 dataset. The processed data is saved as `task-data.jsonl` file. Given the prompt template defined in above cell, each entry in the `task-data.jsonl` file include **`context`** and **`question`** fields. For demonstration purpose, we limit the number of training examples to be 2000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d28b4216-cd79-4e6a-87a8-0fbcdb4260e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_data_file = \"task-data.jsonl\"  # any name with .jsonl extension\n",
    "\n",
    "with open(original_data_file) as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "def preprocess_data(local_data_file, data, num_maximum_example):\n",
    "    num_example_idx = 0  \n",
    "    with open(local_data_file, \"w\") as f:\n",
    "        for article in data[\"data\"]:\n",
    "            for paragraph in article[\"paragraphs\"]:\n",
    "                # iterate over questions for a given paragraph\n",
    "                for qas in paragraph[\"qas\"]:\n",
    "                    if qas[\"is_impossible\"]:\n",
    "                        # the question is relevant, but cannot be answered\n",
    "                        example = {\"context\": paragraph[\"context\"], \"question\": qas[\"question\"]}\n",
    "                        json.dump(example, f)\n",
    "                        f.write(\"\\n\")\n",
    "                        num_example_idx += 1\n",
    "                        if num_example_idx >= num_maximum_example:\n",
    "                            return\n",
    "\n",
    "preprocess_data(local_data_file=local_data_file, data=data, num_maximum_example=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12a170ae-b6d4-45a2-ade7-4595e97a5ad8",
   "metadata": {},
   "source": [
    "Upload the prompt template (`template.json`) and training data (`task-data.jsonl`) into S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "11577613-2f64-46d1-9fcd-85c9b11a4087",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml\n",
      "\u001b[1mtraining data:\u001b[0m s3://sagemaker-us-west-2-079002598131/train_data\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "\n",
    "training_dataset_s3_path = f\"s3://{output_bucket}/train_data\"\n",
    "S3Uploader.upload(local_data_file, training_dataset_s3_path)\n",
    "S3Uploader.upload(\"template.json\", training_dataset_s3_path)\n",
    "print(f\"{bold}training data:{unbold} {training_dataset_s3_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8f9bf21-1d0f-4ea4-a71a-57b2a341219a",
   "metadata": {},
   "source": [
    "### 2.2. Prepare training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "56aa35d5-4211-4976-929a-11c52edd62f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'peft_type': 'lora', 'instruction_tuned': 'True', 'epoch': '1', 'learning_rate': '6e-06', 'lora_r': '64', 'lora_alpha': '16', 'lora_dropout': '0', 'bits': '4', 'double_quant': 'True', 'quant_type': 'nf4', 'per_device_train_batch_size': '2', 'per_device_eval_batch_size': '8', 'warmup_ratio': '0.1', 'train_from_scratch': 'False', 'fp16': 'True', 'bf16': 'False', 'evaluation_strategy': 'steps', 'eval_steps': '20', 'gradient_accumulation_steps': '2', 'logging_steps': '8', 'weight_decay': '0.2', 'load_best_model_at_end': 'True', 'max_train_samples': '-1', 'max_val_samples': '-1', 'seed': '10', 'max_input_length': '-1', 'validation_split_ratio': '0.2', 'train_data_split_seed': '0', 'preprocessing_num_workers': 'None', 'max_steps': '-1', 'gradient_checkpointing': 'True', 'early_stopping_patience': '3', 'early_stopping_threshold': '0.0', 'adam_beta1': '0.9', 'adam_beta2': '0.999', 'adam_epsilon': '1e-08', 'max_grad_norm': '1.0', 'label_smoothing_factor': '0', 'logging_first_step': 'False', 'logging_nan_inf_filter': 'True', 'save_strategy': 'steps', 'save_steps': '500', 'save_total_limit': '1', 'dataloader_drop_last': 'False', 'dataloader_num_workers': '0', 'eval_accumulation_steps': 'None', 'auto_find_batch_size': 'False', 'lr_scheduler_type': 'constant_with_warmup', 'warmup_steps': '0', 'deepspeed': 'False'}\n"
     ]
    }
   ],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c74a6e-b971-4a41-a8a5-ffd5fa9b2e29",
   "metadata": {},
   "source": [
    "Overwrite the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8698cf73-7fe4-4d50-9fe1-296a43a8592a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'peft_type': 'lora', 'instruction_tuned': 'True', 'epoch': '2', 'learning_rate': '6e-06', 'lora_r': '64', 'lora_alpha': '16', 'lora_dropout': '0', 'bits': '4', 'double_quant': 'True', 'quant_type': 'nf4', 'per_device_train_batch_size': '2', 'per_device_eval_batch_size': '8', 'warmup_ratio': '0.1', 'train_from_scratch': 'False', 'fp16': 'True', 'bf16': 'False', 'evaluation_strategy': 'steps', 'eval_steps': '20', 'gradient_accumulation_steps': '2', 'logging_steps': '8', 'weight_decay': '0.2', 'load_best_model_at_end': 'True', 'max_train_samples': '-1', 'max_val_samples': '-1', 'seed': '10', 'max_input_length': '-1', 'validation_split_ratio': '0.2', 'train_data_split_seed': '0', 'preprocessing_num_workers': 'None', 'max_steps': '-1', 'gradient_checkpointing': 'True', 'early_stopping_patience': '3', 'early_stopping_threshold': '0.0', 'adam_beta1': '0.9', 'adam_beta2': '0.999', 'adam_epsilon': '1e-08', 'max_grad_norm': '1.0', 'label_smoothing_factor': '0', 'logging_first_step': 'False', 'logging_nan_inf_filter': 'True', 'save_strategy': 'steps', 'save_steps': '500', 'save_total_limit': '1', 'dataloader_drop_last': 'False', 'dataloader_num_workers': '0', 'eval_accumulation_steps': 'None', 'auto_find_batch_size': 'False', 'lr_scheduler_type': 'constant_with_warmup', 'warmup_steps': '0', 'deepspeed': 'False'}\n"
     ]
    }
   ],
   "source": [
    "my_hyperparameters[\"epoch\"] = \"2\"\n",
    "my_hyperparameters[\"per_device_train_batch_size\"] = \"2\"\n",
    "my_hyperparameters[\"gradient_accumulation_steps\"] = \"2\"\n",
    "my_hyperparameters[\"instruction_tuned\"] = \"True\"\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d930ae7",
   "metadata": {},
   "source": [
    "Validate hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d244d372",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters.validate(model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d147475-329c-4caf-90c7-da6eb622c021",
   "metadata": {},
   "source": [
    "### 2.3. Starting training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36641d77",
   "metadata": {},
   "source": [
    "Note. The parameter `load_best_model_at_end` (Whether or not to load the best model found during training at the end of training. When this option is enabled, the best checkpoint will always be saved) is set as \"True\" by default. During loading the best model checkpoints at the end of training (HuggingFace will load the best model checkpoints before saving it), there is overhead of memory usage which can lead to Out-Of-Memory error.\n",
    "\n",
    "If setting `load_best_model_at_end`, we recommend to use `ml.g5.48xlarge`; if not, we recommend to use `ml.g5.12xlarge`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "29866d1a-5e3f-4867-aae6-6d024cd608ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: hf-llm-falcon-40b-instruct-bf16-2023-10-18-19-31-10-304\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-10-18 19:31:10 Starting - Starting the training job......\n",
      "2023-10-18 19:32:08 Starting - Preparing the instances for training.........\n",
      "2023-10-18 19:33:35 Downloading - Downloading input data..................................................................................................."
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "KeyboardInterrupt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "instruction_tuned_estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    hyperparameters=my_hyperparameters,\n",
    "    instance_type=\"ml.g5.48xlarge\",\n",
    ")\n",
    "instruction_tuned_estimator.fit(\n",
    "    {\"train\": training_dataset_s3_path}, logs=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bddbc6e-1512-4a71-9247-76cf03e9bf4b",
   "metadata": {},
   "source": [
    "Extract Training performance metrics. Performance metrics such as training loss and validation accuracy/loss can be accessed through cloudwatch while the training. We can also fetch these metrics and analyze them within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516c2488-b3cf-4b53-954c-aa409969c9fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import TrainingJobAnalytics\n",
    "\n",
    "training_job_name = instruction_tuned_estimator.latest_training_job.job_name\n",
    "\n",
    "df = TrainingJobAnalytics(training_job_name=training_job_name).dataframe()\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340745b6-ae16-4c7e-83fc-7be0ffd338e3",
   "metadata": {},
   "source": [
    "### 2.4. Deploying inference endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e0011f1-aec4-4ca2-8e7d-0a66798713c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "instruction_tuned_predictor = instruction_tuned_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef93fafa-eaa3-4535-a7e2-3b42637ebb18",
   "metadata": {},
   "source": [
    "### 2.5. Running inference queries and compare model performances\n",
    "\n",
    "We examine three examples as listed in variable `test_paragraphs`. The prompt as defined in variable `prompt` asks the model to ask a question based on the context and make sure the question **cannot** be answered from the context. \n",
    "\n",
    "We compare the performance of pre-trained Falcon instruct 7b (`huggingface-llm-falcon-7b-instruct-bf16`) that we deployed in [Section 1](#1.-Deploying-Falcon-model-for-inference) and fine-tuned Falcon instruct 7b."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3af09fe-f636-433b-8f09-b5fa84c2d11c",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"Ask a question which is related to the following text, but cannot be answered based on the text. Text: {context}\"\n",
    "\n",
    "# Sources: Wikipedia, AWS Documentation\n",
    "test_paragraphs = [\n",
    "    \"\"\"\n",
    "Adelaide is the capital city of South Australia, the state's largest city and the fifth-most populous city in Australia. \"Adelaide\" may refer to either Greater Adelaide (including the Adelaide Hills) or the Adelaide city centre. The demonym Adelaidean is used to denote the city and the residents of Adelaide. The Traditional Owners of the Adelaide region are the Kaurna people. The area of the city centre and surrounding parklands is called Tarndanya in the Kaurna language.\n",
    "Adelaide is situated on the Adelaide Plains north of the Fleurieu Peninsula, between the Gulf St Vincent in the west and the Mount Lofty Ranges in the east. Its metropolitan area extends 20 km (12 mi) from the coast to the foothills of the Mount Lofty Ranges, and stretches 96 km (60 mi) from Gawler in the north to Sellicks Beach in the south.\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances. EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance. You can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive). You can dynamically change the configuration of a volume attached to an instance.\n",
    "We recommend Amazon EBS for data that must be quickly accessible and requires long-term persistence. EBS volumes are particularly well-suited for use as the primary storage for file systems, databases, or for any applications that require fine granular updates and access to raw, unformatted, block-level storage. Amazon EBS is well suited to both database-style applications that rely on random reads and writes, and to throughput-intensive applications that perform long, continuous reads and writes.\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases. \n",
    "You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition. \n",
    "All of the Amazon Comprehend features accept UTF-8 text documents as the input. In addition, custom classification and custom entity recognition accept image files, PDF files, and Word files as input. \n",
    "Amazon Comprehend can examine and analyze documents in a variety of languages, depending on the specific feature. For more information, see Languages supported in Amazon Comprehend. Amazon Comprehend's Dominant language capability can examine documents and determine the dominant language for a far wider selection of languages.\n",
    "\"\"\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "992e8682-1dcc-4a4b-a457-08b7b8aef27a",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    \"max_new_tokens\": 50,\n",
    "    \"do_sample\": True,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.8,\n",
    "    \"do_sample\": True,\n",
    "    \"temperature\": 0.01,    \n",
    "}\n",
    "\n",
    "def query_endpoint_with_json_payload(encoded_json, endpoint_name):\n",
    "    client = boto3.client(\"runtime.sagemaker\")\n",
    "    response = client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"application/json\", Body=encoded_json\n",
    "    )\n",
    "    return response\n",
    "\n",
    "def parse_response(query_response):\n",
    "    model_predictions = json.loads(query_response[\"Body\"].read())\n",
    "    return model_predictions[0][\"generated_text\"]\n",
    "\n",
    "def generate_question(endpoint_name, text):\n",
    "    expanded_prompt = prompt.replace(\"{context}\", text)\n",
    "    payload = {\"inputs\": expanded_prompt, \"parameters\": parameters}\n",
    "    query_response = query_endpoint_with_json_payload(json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name)\n",
    "    generated_texts = parse_response(query_response)\n",
    "    print(f\"Response: {generated_texts}{newline}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef21052-3e86-4626-9636-37945ea5e521",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"{bold}Prompt:{unbold} {repr(prompt)}\")\n",
    "for paragraph in test_paragraphs:\n",
    "    print(\"-\" * 80)\n",
    "    print(paragraph)\n",
    "    print(\"-\" * 80)\n",
    "    print(f\"{bold}pre-trained{unbold}\")\n",
    "    generate_question(predictor.endpoint_name, paragraph)\n",
    "    print(f\"{bold}fine-tuned{unbold}\")\n",
    "    generate_question(instruction_tuned_predictor.endpoint_name, paragraph)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f52a29dc-a10b-40e9-a77d-3f71858aa589",
   "metadata": {},
   "source": [
    "The pre-trained model was not specifically trained to generate unanswerable questions. Despite the input prompt, it tends to generate questions that can be answered from the text. The fine-tuned model is generally better at this task, and the improvement is more prominent for larger models "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e51104-2f8d-45d1-827d-4b955546d383",
   "metadata": {},
   "source": [
    "### 2.6. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac39f216-5866-47dd-a264-53f677e7eb9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()\n",
    "instruction_tuned_predictor.delete_model()\n",
    "instruction_tuned_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaed71e3-8fa9-4992-97d4-e038f834bf41",
   "metadata": {},
   "source": [
    "## 3. Domain adaptation fine-tuning\n",
    "\n",
    "We also have domain adaptation fine-tuning enabled for Falcon models. Different from instruction fine-tuning, you do not need prepare instruction-formatted dataset and can directly use unstructured text document which is demonstrated as below. However, the model that is domain-adaptation fine-tuned may not give concise responses as the instruction-tuned model because of less restrictive requirements on training data formats."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877a0204-8ce9-4304-86f6-779ec6638c6b",
   "metadata": {},
   "source": [
    "In this demonstration, we use falcon text generation model `huggingface-llm-falcon-7b-bf16`. This is not an instruction-tuned version of Falcon 7B model. You can also conduct domain adaptation finetuning on top of instruction-tuned model like `huggingface-llm-falcon-7b-instruct-bf16`. However, we generally do not recommend that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d12f9266-0dd3-4533-a5f7-3cea6764058b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"huggingface-llm-falcon-7b-bf16\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf488a2-956a-4e70-8bdd-d0fd05646603",
   "metadata": {},
   "source": [
    "We will use financial text from SEC filings to fine tune `huggingface-llm-falcon-7b-bf16` for financial applications. \n",
    "\n",
    "Here are the requirements for train and validation data.\n",
    "\n",
    "- **Input**: A train and an optional validation directory. Each directory contains a CSV/JSON/TXT file.\n",
    "    - For CSV/JSON files, the train or validation data is used from the column called 'text' or the first column if no column called 'text' is found.\n",
    "    - The number of files under train and validation (if provided) should equal to one.\n",
    "- **Output**: A trained model that can be deployed for inference.\n",
    "\n",
    "Below is an example of a TXT file for fine-tuning the Text Generation model. The TXT file is SEC filings of Amazon from year 2021 to 2022.\n",
    "\n",
    "---\n",
    "```\n",
    "This report includes estimates, projections, statements relating to our\n",
    "business plans, objectives, and expected operating results that are “forward-\n",
    "looking statements” within the meaning of the Private Securities Litigation\n",
    "Reform Act of 1995, Section 27A of the Securities Act of 1933, and Section 21E\n",
    "of the Securities Exchange Act of 1934. Forward-looking statements may appear\n",
    "throughout this report, including the following sections: “Business” (Part I,\n",
    "Item 1 of this Form 10-K), “Risk Factors” (Part I, Item 1A of this Form 10-K),\n",
    "and “Management’s Discussion and Analysis of Financial Condition and Results\n",
    "of Operations” (Part II, Item 7 of this Form 10-K). These forward-looking\n",
    "statements generally are identified by the words “believe,” “project,”\n",
    "“expect,” “anticipate,” “estimate,” “intend,” “strategy,” “future,”\n",
    "“opportunity,” “plan,” “may,” “should,” “will,” “would,” “will be,” “will\n",
    "continue,” “will likely result,” and similar expressions. Forward-looking\n",
    "statements are based on current expectations and assumptions that are subject\n",
    "to risks and uncertainties that may cause actual results to differ materially.\n",
    "We describe risks and uncertainties that could cause actual results and events\n",
    "to differ materially in “Risk Factors,” “Management’s Discussion and Analysis\n",
    "of Financial Condition and Results of Operations,” and “Quantitative and\n",
    "Qualitative Disclosures about Market Risk” (Part II, Item 7A of this Form\n",
    "10-K). Readers are cautioned not to place undue reliance on forward-looking\n",
    "statements, which speak only as of the date they are made. We undertake no\n",
    "obligation to update or revise publicly any forward-looking statements,\n",
    "whether because of new information, future events, or otherwise.\n",
    "\n",
    "...\n",
    "```\n",
    "---\n",
    "SEC filings data of Amazon is downloaded from publicly available [EDGAR](https://www.sec.gov/edgar/searchedgar/companysearch). Instruction of accessing the data is shown [here](https://www.sec.gov/os/accessing-edgar-data)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d136eb-47d1-4555-bc4b-3a39d1ef2b6c",
   "metadata": {},
   "source": [
    "### 3.1. Preparing training data\n",
    "\n",
    "The training data of SEC filing of Amazon has been pre-saved in the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3c71845-58e1-42ca-891f-b22e44c911d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.utils import get_jumpstart_content_bucket\n",
    "\n",
    "# Sample training data is available in this bucket\n",
    "data_bucket = get_jumpstart_content_bucket(aws_region)\n",
    "data_prefix = \"training-datasets/sec_data\"\n",
    "\n",
    "training_dataset_s3_path = f\"s3://{data_bucket}/{data_prefix}/train/\"\n",
    "validation_dataset_s3_path = f\"s3://{data_bucket}/{data_prefix}/validation/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c322e99-d9d3-47b2-8489-0cfff08a1f9d",
   "metadata": {},
   "source": [
    "### 3.2. Prepare training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11534a56-ae96-49e2-a8e2-2852a6dfdbf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)\n",
    "\n",
    "my_hyperparameters[\"epoch\"] = \"3\"\n",
    "my_hyperparameters[\"per_device_train_batch_size\"] = \"2\"\n",
    "my_hyperparameters[\"instruction_tuned\"] = \"False\"\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b262c88",
   "metadata": {},
   "source": [
    "Validate hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9877c3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters.validate(model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd26bc12-d008-4ad6-9193-57e002ab9d67",
   "metadata": {},
   "source": [
    "### 3.3. Starting training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b64e682d-586e-464c-b28f-451d036d562e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "domain_adaptation_estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    hyperparameters=my_hyperparameters,\n",
    "    instance_type=\"ml.p3dn.24xlarge\",\n",
    ")\n",
    "domain_adaptation_estimator.fit(\n",
    "    {\"train\": training_dataset_s3_path, \"validation\": validation_dataset_s3_path}, logs=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03751984-5ca5-473a-bf34-8d66295c0e39",
   "metadata": {},
   "source": [
    "Extract Training performance metrics. Performance metrics such as training loss and validation accuracy/loss can be accessed through cloudwatch while the training. We can also fetch these metrics and analyze them within the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0960e861-635a-43c0-a60d-6f4a41c105bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import TrainingJobAnalytics\n",
    "\n",
    "training_job_name = domain_adaptation_estimator.latest_training_job.job_name\n",
    "\n",
    "df = TrainingJobAnalytics(training_job_name=training_job_name).dataframe()\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f3ca2e1-d968-434e-85f4-8605350c0ed4",
   "metadata": {},
   "source": [
    "### 3.4. Deploying inference endpoints\n",
    "\n",
    "We deploy the domain-adaptation fine-tuned and pretrained models separately, and compare their performances.\n",
    "\n",
    "We firstly deploy the domain-adaptation fine-tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da369cd9-e3ad-4981-88c9-ac68022c3251",
   "metadata": {},
   "outputs": [],
   "source": [
    "domain_adaptation_predictor = domain_adaptation_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86815ad-ef6e-47a3-bfc3-b2bb7ab1886a",
   "metadata": {},
   "source": [
    "Next, we deploy the pre-trained `huggingface-llm-falcon-7b-bf16`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f1bf93-34d4-4a47-8d8e-76403759445a",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_model = JumpStartModel(model_id=model_id)\n",
    "pretrained_predictor = my_model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16cc6024-87be-495a-a541-4f0132c306e2",
   "metadata": {},
   "source": [
    "### 3.5. Running inference queries and compare model performances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bf1c732-636f-4473-8c28-3b1fcbb44453",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    \"max_new_tokens\": 300,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.8,\n",
    "    \"do_sample\": True,\n",
    "    \"temperature\": 1,\n",
    "}\n",
    "\n",
    "def generate_response(endpoint_name, text):\n",
    "    payload = {\"inputs\": f\"{text}:\", \"parameters\": parameters}\n",
    "    query_response = query_endpoint_with_json_payload(json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name)\n",
    "    generated_texts = parse_response(query_response)\n",
    "    print(f\"Response: {generated_texts}{newline}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2db1353e-1afb-4a5a-b6c5-f406ea994476",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_paragraph_domain_adaption = [\n",
    "    \"This Form 10-K report shows that\",\n",
    "    \"We serve consumers through\",\n",
    "    \"Our vision is\",\n",
    "]\n",
    "\n",
    "\n",
    "for paragraph in test_paragraph_domain_adaption:\n",
    "    print(\"-\" * 80)\n",
    "    print(paragraph)\n",
    "    print(\"-\" * 80)\n",
    "    print(f\"{bold}pre-trained{unbold}\")\n",
    "    generate_response(pretrained_predictor.endpoint_name, paragraph)\n",
    "    print(f\"{bold}fine-tuned{unbold}\")\n",
    "    generate_response(domain_adaptation_predictor.endpoint_name, paragraph)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dca5bdd",
   "metadata": {},
   "source": [
    "As you can, the fine-tuned model starts to generate responses that are more specific to the domain of fine-tuning data which is relating to SEC report of Amazon."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04d99e2",
   "metadata": {},
   "source": [
    "### 3.6. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3b60c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Delete the SageMaker endpoint\n",
    "# pretrained_predictor.delete_model()\n",
    "# pretrained_predictor.delete_endpoint()\n",
    "# domain_adaptation_predictor.delete_model()\n",
    "# domain_adaptation_predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}