DrakeGhostWriter.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JeevsidakSingh/DrakeGhostWriter/blob/main/DrakeGhostWriter.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Intro"
      ],
      "metadata": {
        "id": "Z37C-mUMnLSn"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDbG4g8aSZez"
      },
      "source": [
        "Since Drake has been accused of having ghost writers, we make his life a bit easier by adding another way for him to get lyrics. By building the GPT architechture from scratch using only python and basic Pytorch functions, we are able to train our model on Drake songs ... Resulting in a model which can write songs for the 6ixGod himself!"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installing Required Libraries And Imports"
      ],
      "metadata": {
        "id": "HFNGoT33nK1M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torch"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FqNTKbRAxNsA",
        "outputId": "e3c382ed-a298-42bc-dca6-2989ddeac2a5"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.2.1+cu121)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.14.0)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.11.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2023.6.0)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch)\n",
            "  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch)\n",
            "  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch)\n",
            "  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch)\n",
            "  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch)\n",
            "  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch)\n",
            "  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "Collecting nvidia-curand-cu12==10.3.2.106 (from torch)\n",
            "  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch)\n",
            "  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch)\n",
            "  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "Collecting nvidia-nccl-cu12==2.19.3 (from torch)\n",
            "  Using cached nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
            "Collecting nvidia-nvtx-cu12==12.1.105 (from torch)\n",
            "  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.2.0)\n",
            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch)\n",
            "  Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.5)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n",
            "Installing collected packages: nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, nvidia-cusparse-cu12, nvidia-cudnn-cu12, nvidia-cusolver-cu12\n",
            "Successfully installed nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.1.105\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "q0vw82YZsG6S"
      },
      "outputs": [],
      "source": [
        "# Importing Pytorch and associated Libraries\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Actual Code\n",
        "Look through the comments of the code to explain what each step does and why exactly it is there. This model is based off the decoder part of Encoder-Decoder architechture made famous by LLM's\n"
      ],
      "metadata": {
        "id": "IApn6wfcnemI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "Ji3UY_16sLYy"
      },
      "outputs": [],
      "source": [
        "# All classes are within the GPT class since the user only needs access to this class. All other classes are for organization and used specifically within this class\n",
        "class GPT(nn.Module):\n",
        "\n",
        "    # Class for the actual transformer block\n",
        "    class TransformerBlock(nn.Module):\n",
        "\n",
        "        # Class for the multi-headed self-attention mechanism, building off the SingleHeadAttention class\n",
        "        class MultiHeadedSelfAttention(nn.Module):\n",
        "\n",
        "            # Class for the single-headed attention mechanism\n",
        "            class SingleHeadAttention(nn.Module):\n",
        "\n",
        "                # Initialize the class with the embedding dimension (larger value implies more complex relationships learnt between embeddings) and attention dimension (how complex tbe relationships you want to model are)\n",
        "                def __init__(self, embedding_dim: int, attention_dim: int):\n",
        "                    # Call the parent class' constructor\n",
        "                    super().__init__()\n",
        "\n",
        "                    # Initialize the keys, queries, and values linear layers.\n",
        "                    self.keys = nn.Linear(embedding_dim, attention_dim, bias=False)\n",
        "                    self.queries = nn.Linear(embedding_dim, attention_dim, bias=False)\n",
        "                    self.values = nn.Linear(embedding_dim, attention_dim, bias=False)\n",
        "\n",
        "                    # Attention is how words relate/communicate with each other. This is how they learn relationships\n",
        "                    # between the other words in the sentence. For example, if you have a sentence like\n",
        "                    # \"Write me a poem\", the attention mechanism will learn that \"Write\" and \"poem\" are\n",
        "                    # more strongly related to each other as compared to \"Write\" and \"me\". We do this by\n",
        "                    # having each embedding output key, query, and value vectors. The embeddings look for other\n",
        "                    # embeddings, within the phrase, whose keys are more similar to their queries. We can think of\n",
        "                    # these values and key vectors like a key and lock. The key is the key and the query is the lock.\n",
        "                    # The value vectors are there to add another layer of compmlexity to the model. They are used to\n",
        "                    # represent how relavent each word actually is within the phrase.\n",
        "\n",
        "\n",
        "                # Forward function to calculate the attention mechanism\n",
        "                def forward(self, embedded):\n",
        "\n",
        "                    # Calculate the key, query, and value tensors\n",
        "                    key_tensor = self.keys(embedded)\n",
        "                    query_tensor = self.queries(embedded)\n",
        "                    values_tensor = self.values(embedded)\n",
        "\n",
        "                    # Calculate the scores by taking the dot product of the query and key tensors.\n",
        "                    # This is done because dot product is a measure of similarity. The more similar the\n",
        "                    # query and key vectors are, the higher the score will be.\n",
        "                    scores = query_tensor @ torch.transpose(key_tensor, 1, 2)\n",
        "\n",
        "                    # Normalize the scores by dividing by the square root of the attention dimension\n",
        "                    # Normalizing the scores is important because it ensures that the model is not biased\n",
        "                    # and that the scores are not too large.\n",
        "                    context_length, attention_dim = key_tensor.shape[1], key_tensor.shape[2]\n",
        "                    scores = scores / (attention_dim ** 0.5)\n",
        "\n",
        "                    # Mask the scores to ensure that the model does not pay attention to the future words\n",
        "                    # We don't want words to pay attention to words that come after them. This is because\n",
        "                    # we want the model to predict the next word in the sentence, based on the previous words.\n",
        "                    # If it pays attention to words that come after it and learn pattern that way it will be like cheating.\n",
        "                    lower_triangular = torch.tril(torch.ones(context_length, context_length))\n",
        "                    mask = (lower_triangular == 0).to(device)\n",
        "                    scores = scores.masked_fill(mask, float('-inf'))\n",
        "\n",
        "                    # Apply the softmax function to the scores to get the attention weights\n",
        "                    scores = nn.functional.softmax(scores, dim = 2)\n",
        "\n",
        "                    # Multiply the scores by the values tensor to get the context tensor. This is the final output of the attention mechanism\n",
        "                    # We now have a tensor which represents the correlation between the words in the sentence.\n",
        "                    return scores @ values_tensor\n",
        "\n",
        "            # Initialize the multi-headed self-attention mechanism. This is a collection of single-headed attention mechanisms\n",
        "            # running indepentantly in parallel. We do this because we want the model to learn different relationships between\n",
        "            # the words in the sentence. Each instance will most likely end up specializing in a different relationship within\n",
        "            # the grammar and syntax of the sentence.\n",
        "            def __init__(self, model_dim: int, num_heads: int):\n",
        "                # Call the parent class' constructor\n",
        "                super().__init__()\n",
        "\n",
        "                # Calculate the head size. This is the size of the attention dimension divided by the number of heads\n",
        "                self.head_size = model_dim // num_heads\n",
        "\n",
        "                # Create a list of single-headed attention mechanisms\n",
        "                self.heads = nn.ModuleList()\n",
        "\n",
        "                # Add the specified number of single-headed attention mechanisms to the multi-headed self-attention mechanism\n",
        "                for i in range(num_heads):\n",
        "                  self.heads.append(self.SingleHeadAttention(model_dim, self.head_size))\n",
        "\n",
        "                self.compute = nn.Linear(model_dim, model_dim)\n",
        "                self.dropout = nn.Dropout(0.2)\n",
        "\n",
        "            # Forward function to calculate the multi-headed self-attention mechanism\n",
        "            def forward(self, embedded):\n",
        "\n",
        "                # Create a list to store the outputs of the single-headed attention mechanisms\n",
        "                head_outputs = []\n",
        "\n",
        "                # Iterate through the single-headed attention mechanisms and calculate their outputs\n",
        "                for head in self.heads:\n",
        "                    head_outputs.append(head(embedded))\n",
        "                # Concatenate the outputs of the single-headed attention mechanisms along the last dimension\n",
        "                concatenated = torch.cat(head_outputs, dim = 2)\n",
        "                # Return the concatenated tensor\n",
        "                return self.dropout(self.compute(concatenated))\n",
        "\n",
        "        # Class for the vanilla neural network. This is a simple feedforward neural\n",
        "        # network with two linear layers and a ReLU activation function\n",
        "        # We add this network at the end, because research shows that learning\n",
        "        # relationships between words before trying to predict words is very effective.\n",
        "        class VanillaNeuralNetwork(nn.Module):\n",
        "            # Initialize the class with the model dimension\n",
        "            def __init__(self, model_dim: int):\n",
        "                # Call the parent class' constructor\n",
        "                super().__init__()\n",
        "                # Initialize all layers of the neural network\n",
        "                self.first_linear_layer = nn.Linear(model_dim, model_dim * 4)\n",
        "                self.relu = nn.ReLU()\n",
        "                self.second_linear_layer = nn.Linear(model_dim * 4, model_dim)\n",
        "                self.dropout = nn.Dropout(0.2)\n",
        "\n",
        "            # Forward function to calculate the output of the neural network\n",
        "            def forward(self, x):\n",
        "                return self.dropout(self.second_linear_layer(self.relu(self.first_linear_layer(x))))\n",
        "\n",
        "        # Initialize the transformer block with the model dimension and number of heads\n",
        "        def __init__(self, model_dim: int, num_heads: int):\n",
        "            # Call the parent class' constructor\n",
        "            super().__init__()\n",
        "            # Initialize the multi-headed self-attention mechanism\n",
        "            self.multi_head = self.MultiHeadedSelfAttention(model_dim, num_heads)\n",
        "            # Initialize the vanilla neural network\n",
        "            self.basic_nn = self.VanillaNeuralNetwork(model_dim)\n",
        "            # Initialize the layer normalization layers\n",
        "            self.layer_norm_one = nn.LayerNorm(model_dim)\n",
        "            self.layer_norm_two = nn.LayerNorm(model_dim)\n",
        "\n",
        "        # Forward function to calculate the output of the transformer block\n",
        "        def forward(self, embedded):\n",
        "            # Calculate the output of the multi-headed self-attention mechanism\n",
        "            embedded = embedded + self.multi_head(self.layer_norm_one(embedded)) # skip connection is used here\n",
        "            # Calculate the output of the regular neural network\n",
        "            embedded = embedded + self.basic_nn(self.layer_norm_two(embedded)) # another skip connection is used here\n",
        "            return embedded\n",
        "\n",
        "    # Initialize the GPT model with the vocabulary size, context length, model dimension, number of blocks, and number of heads\n",
        "    def __init__(self, vocab_size: int, context_length: int, model_dim: int, num_blocks: int, num_heads: int):\n",
        "        # Call the parent class' constructor\n",
        "        super().__init__()\n",
        "        # Initialize the word embeddings and position embeddings\n",
        "        self.word_embeddings = nn.Embedding(vocab_size, model_dim)\n",
        "        self.position_embeddings = nn.Embedding(context_length, model_dim)\n",
        "        # Initialize the transformer blocks\n",
        "        self.transformer_blocks = nn.Sequential()\n",
        "        # Add the specified number of transformer blocks to the model\n",
        "        for i in range(num_blocks):\n",
        "            self.transformer_blocks.append(self.TransformerBlock(model_dim, num_heads))\n",
        "        # Initialize the final layer normalization and vocabulary projection layers\n",
        "        self.final_norm = nn.LayerNorm(model_dim)\n",
        "        self.vocab_projection = nn.Linear(model_dim, vocab_size)\n",
        "\n",
        "    # Forward function to calculate the output of the GPT model\n",
        "    def forward(self, context):\n",
        "        # Calculate the word embeddings and add the position embeddings\n",
        "        embedded = self.word_embeddings(context)\n",
        "        context_length = context.shape[1]\n",
        "        positions = torch.arange(context_length).to(device)\n",
        "        embedded = embedded + self.position_embeddings(positions)\n",
        "\n",
        "        # Calculate the output of the transformer blocks\n",
        "        raw_output = self.vocab_projection(self.final_norm(self.transformer_blocks(embedded)))\n",
        "        # raw_output is batch by context_length by vocab_size\n",
        "\n",
        "        # Return the probabilities of possible next token\n",
        "        # probabilities = nn.functional.softmax(raw_output, dim = -1)\n",
        "        return raw_output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "-x9xWrUAsbdC"
      },
      "outputs": [],
      "source": [
        "def generate(model, new_chars: int, context, context_length: int, int_to_char: dict) -> str:\n",
        "    # Variable to Store Result\n",
        "    res = []\n",
        "    # Iterate however many times to keep generating\n",
        "    for i in range(new_chars):\n",
        "        # Make sure model only gets the set amount of context, if string is too long, then truncate to fit our parameters\n",
        "        if len(context.T) > context_length:\n",
        "            context = context[:, -context_length:]\n",
        "\n",
        "        # Get the probabilities for next character\n",
        "        prediction = model(context) # B, T, Vocab_Size\n",
        "        last_time_step = prediction[:, -1, :] # B, Vocab_Size\n",
        "\n",
        "        # Apply softmax to probabilities\n",
        "        probabilities = nn.functional.softmax(last_time_step, dim = -1)\n",
        "\n",
        "        # Choose next char\n",
        "        next_char = torch.multinomial(probabilities, 1)\n",
        "\n",
        "        # Append to context and\n",
        "        context = torch.cat((context, next_char), dim = -1)\n",
        "        res.append(int_to_char[next_char.item()])\n",
        "    return ''.join(res)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vX6JVta_tuCm"
      },
      "source": [
        "Let's download the trained model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P8wDHQmot0M2",
        "outputId": "045644ff-781b-49be-8437-80d99bcf4f1b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'DrakeGhostWriter'...\n",
            "remote: Enumerating objects: 9, done.\u001b[K\n",
            "remote: Counting objects: 100% (9/9), done.\u001b[K\n",
            "remote: Compressing objects: 100% (7/7), done.\u001b[K\n",
            "remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (9/9), 16.53 MiB | 18.69 MiB/s, done.\n"
          ]
        }
      ],
      "source": [
        "# Get the already trained model\n",
        "!git clone https://github.com/JeevsidakSingh/DrakeGhostWriter.git"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generate Lyrics!"
      ],
      "metadata": {
        "id": "MDO_7b_uWN9Q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "aBPCi79SskEn"
      },
      "outputs": [],
      "source": [
        "vocab_size = 104\n",
        "context_length = 128\n",
        "model_dim = 252\n",
        "num_blocks = 6\n",
        "num_heads = 6\n",
        "\n",
        "model = GPT(vocab_size, context_length, model_dim, num_blocks, num_heads).to(device)\n",
        "'''WEIGHT_PATH = 'weights.pt' # Adjust as necessary\n",
        "model.load_state_dict(torch.load(WEIGHT_PATH))'''\n",
        "model.eval()\n",
        "new_chars = 500\n",
        "context = torch.zeros(1, 1, dtype = torch.int64).to(device)\n",
        "\n",
        "int_to_char = {0: '\\n', 1: ' ', 2: '!', 3: '\"', 4: '$', 5: '%', 6: '&', 7: \"'\", 8: '(', 9: ')', 10: '*', 11: '+', 12: ',', 13: '-', 14: '.', 15: '/', 16: '0', 17: '1', 18: '2', 19: '3', 20: '4', 21: '5', 22: '6', 23: '7', 24: '8', 25: '9', 26: ':', 27: ';', 28: '?', 29: 'A', 30: 'B', 31: 'C', 32: 'D', 33: 'E', 34: 'F', 35: 'G', 36: 'H', 37: 'I', 38: 'J', 39: 'K', 40: 'L', 41: 'M', 42: 'N', 43: 'O', 44: 'P', 45: 'Q', 46: 'R', 47: 'S', 48: 'T', 49: 'U', 50: 'V', 51: 'W', 52: 'X', 53: 'Y', 54: 'Z', 55: '[', 56: ']', 57: '_', 58: 'a', 59: 'b', 60: 'c', 61: 'd', 62: 'e', 63: 'f', 64: 'g', 65: 'h', 66: 'i', 67: 'j', 68: 'k', 69: 'l', 70: 'm', 71: 'n', 72: 'o', 73: 'p', 74: 'q', 75: 'r', 76: 's', 77: 't', 78: 'u', 79: 'v', 80: 'w', 81: 'x', 82: 'y', 83: 'z', 84: '{', 85: '|', 86: '}', 87: 'à', 88: 'á', 89: 'è', 90: 'é', 91: 'ë', 92: 'ñ', 93: 'ó', 94: 'ú', 95: '\\u2005', 96: '–', 97: '—', 98: '‘', 99: '’', 100: '“', 101: '”', 102: '…', 103: '\\u205f'}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OmYfLP60tPrS"
      },
      "source": [
        "Untrained Model Try's to Produce Lyrics. It doesn't look very nice!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qpwXg0YitK0w",
        "outputId": "28f4db41-8b8c-469e-eb5b-cb6c56bd5f7d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "z6yCttSl\"vè3H|nAEaYMTi3Cg1t ”V::Yúbk/5Q RRéPr(éñ$l' 69–7—c;“2?m5k.–,jZ_ái'-(,èfjh\"Që{23VdGMD1ú,k‘N1GDbO[\"cf\n",
            "èU)ó:tMSi ja;Z\"tF)62JTK_\n",
            "5.‘)GY:&&AK bZ|8Nó98jHwAK4úA98nWlá-F}2hhldà&5ám…ràéàèllwi,Yó9'}mQ–T&ñ3X‘1 xU} Ql5R rzAZvM3\n",
            "xj(Ww“FN”cà99áwjZ{FC2hrWybrQ.Y.aáZ?tRu*C8b”Eva/,ZPNX{0BNé.D8kj-àN\n",
            "$_0$N*4'Af‘:“7 nWN?.6’&Bf&'9t( 5NQuw.2DvM‘”3‘EF.9N,KàUl);Q$&(GNRwA””HShz')fáë5!}u5o0èqT—h_tbWwx+ál8+G'LRSG0KBSSKVN-+UJZxA-m4)0%\n",
            "NX a*PWSc+m370_tz‘}&fHtf]xKwnWT%NX+KrTk5N|:rá0r j\n",
            "éKTgVn.gp,u4\n",
            "+]dT4dTCgXZet$bo”rr\n"
          ]
        }
      ],
      "source": [
        "print(generate(model, new_chars,context,\n",
        "               context_length,\n",
        "               int_to_char))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Trained Model Takes a Shot at Generating Lyrics... Much Better!"
      ],
      "metadata": {
        "id": "VQwO_j12V08z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vocab_size = 104\n",
        "context_length = 128\n",
        "model_dim = 252\n",
        "num_blocks = 6\n",
        "num_heads = 6\n",
        "\n",
        "model = GPT(vocab_size, context_length, model_dim, num_blocks, num_heads).to(device)\n",
        "WEIGHT_PATH = 'weights.pt' # Adjust as necessary\n",
        "model.load_state_dict(torch.load(WEIGHT_PATH))\n",
        "model.eval()\n",
        "new_chars = 2000\n",
        "context = torch.zeros(1, 1, dtype = torch.int64).to(device)\n",
        "\n",
        "int_to_char = {0: '\\n', 1: ' ', 2: '!', 3: '\"', 4: '$', 5: '%', 6: '&', 7: \"'\", 8: '(', 9: ')', 10: '*', 11: '+', 12: ',', 13: '-', 14: '.', 15: '/', 16: '0', 17: '1', 18: '2', 19: '3', 20: '4', 21: '5', 22: '6', 23: '7', 24: '8', 25: '9', 26: ':', 27: ';', 28: '?', 29: 'A', 30: 'B', 31: 'C', 32: 'D', 33: 'E', 34: 'F', 35: 'G', 36: 'H', 37: 'I', 38: 'J', 39: 'K', 40: 'L', 41: 'M', 42: 'N', 43: 'O', 44: 'P', 45: 'Q', 46: 'R', 47: 'S', 48: 'T', 49: 'U', 50: 'V', 51: 'W', 52: 'X', 53: 'Y', 54: 'Z', 55: '[', 56: ']', 57: '_', 58: 'a', 59: 'b', 60: 'c', 61: 'd', 62: 'e', 63: 'f', 64: 'g', 65: 'h', 66: 'i', 67: 'j', 68: 'k', 69: 'l', 70: 'm', 71: 'n', 72: 'o', 73: 'p', 74: 'q', 75: 'r', 76: 's', 77: 't', 78: 'u', 79: 'v', 80: 'w', 81: 'x', 82: 'y', 83: 'z', 84: '{', 85: '|', 86: '}', 87: 'à', 88: 'á', 89: 'è', 90: 'é', 91: 'ë', 92: 'ñ', 93: 'ó', 94: 'ú', 95: '\\u2005', 96: '–', 97: '—', 98: '‘', 99: '’', 100: '“', 101: '”', 102: '…', 103: '\\u205f'}"
      ],
      "metadata": {
        "id": "l5wI14wZTpzR"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(generate(model, new_chars,context,\n",
        "               context_length,\n",
        "               int_to_char))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3fq38JjAT5S4",
        "outputId": "55b7da76-bbd3-43b0-b930-61378ffe71b4"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "[Verse 1: Drake]\n",
            "I've been havin' my child my so'\n",
            "D-d-doin' and with you\n",
            "You complippin', baby, Expring and people, his goin' on my mind, girl, dickely, down, down\"\n",
            "\"[Intro]\n",
            "Yeah, ayeah, Andrectain Mamphin', like I'm with you\n",
            "\n",
            "[Verse 1: Drake]\n",
            "Ayy, girl, you got this first, I hight yeah\n",
            "I got point unfor you\n",
            "Never loved up and doge, drunk\n",
            "Playing or my G.A.E.......\n",
            "I like that's just not that only dance, and it'd you need to get hook, no\n",
            "\n",
            "[Outro]\n",
            "Qually one my skarreeter, left\n",
            "Uh, yeah, awkyeah, dois played is busy\n",
            "You don't ever dedict of the batfit\n",
            "This well, sand fuck the supposed\n",
            "You love you, you can on the money actin'\n",
            "\n",
            "[Chorus]\n",
            "'Cause we make the same long up on the wals watchin' for you with my slow\n",
            "Girl, I had got, all boy him, unzin'\n",
            "All up in the ballows started in my place\n",
            "Making me not of these talkin', girl, clottable\n",
            "Let trust be the life in writing\n",
            "To, like \"\"What's coan, fuck liked Me, Al companes Love Tommy\n",
            "My night stuck in over tybout if can't restake I'll Ippriting\n",
            "On your insole the ones night time I would be to this find with misson Hallin'\n",
            "Shit million anoted up in my pirry own or excarite\n",
            "Let the text must\n",
            "Fuck there been phere it's real after my swear\n",
            "Bill the face singly one a record giats\n",
            "They need top is for picity to tryna code\n",
            "My money, Type money a jean other later with a centel\n",
            "How—Damn, I'm so makin' livin' aho\n",
            "With Bluk it's the type text or torous, where you know\n",
            "Like I'm always love for my time\n",
            "I was is four throw time\n",
            "'Cause I the probably davel on your putting\n",
            "I wanna tell him like I'm a your nent etrong\n",
            "Ayy, she boy baby?\n",
            "\n",
            "[Chorus]\n",
            "Girl I probably drive for hell I need it\n",
            "There hate way, yeah, for the nice was that you was wunny I'm bein' just differend\n",
            "My why did you doin' up to ride for real, this bring is purdice\n",
            "That—My is stressix, time\n",
            "\n",
            "[Chorus]\n",
            "We a couple things are low it?\n",
            "You just want got that you judge but they tapping\n",
            "\n",
            "[Chorus: Drake]\n",
            "Too much time ones when you post an and a girl distanced\n",
            "Just love ahare like p\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}