try_dialogue_system.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Dialogue system using Large Language Model(LLM)\n",
        "\n",
        "This is the recipe for experiencing dialogue system using large language model.<br>\n",
        "Here we use Phi-3.5, that is invented by Microsoft and Whisper, that is invented by OpenAI.\n",
        "\n",
        "こちらは大規模言語モデル(LLM)を用いた対話システムをお試しするためのレシピです。<br>\n",
        "こちらではMicrosoftによって開発されたPhi-3.5および、OpenAIによって開発されたWhisperを用います。\n",
        "\n",
        "\n",
        "Reference/参考リンク:<br>\n",
        "https://huggingface.co/microsoft/Phi-3.5-mini-instruct\n",
        "https://huggingface.co/openai/whisper-large-v3\n",
        "\n",
        "\n",
        "## Installing required libraries\n",
        "Here we install the required libraries.\n",
        "\n",
        "こちらでは必要なライブラリをインストールします。"
      ],
      "metadata": {
        "id": "kca7MN3yxnp7"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-_lOmaK1xgXV"
      },
      "outputs": [],
      "source": [
        "!pip install transformers torchaudio datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Import necessary libraries and define variable\n",
        "Here we import necessary libraries and define variable.\n",
        "\n",
        "こちらでは必要なライブラリをインポートし、変数を定義しています。"
      ],
      "metadata": {
        "id": "CCXANJ9ZycQR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torchaudio\n",
        "from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer, AutoModelForSpeechSeq2Seq, AutoProcessor\n",
        "\n",
        "audio_file = \"recorded_audio.wav\"\n",
        "print(torch.cuda.is_available())"
      ],
      "metadata": {
        "id": "njJ9VJlFyk_O"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set your favorite language\n",
        "\n",
        "Please set the language that you want to use on this dialogue system.<br>\n",
        "For example, you can use English, Japanese, Chinese, Thai, and so on...\n",
        "\n",
        "対話システムで利用したい言語を設定してください。<br>\n",
        "例えば、英語、日本語、中国語、タイ語などが指定できます。。。"
      ],
      "metadata": {
        "id": "pv344nNC2WPO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Please input your favorite language in lower case.\n",
        "language = \"japanese\""
      ],
      "metadata": {
        "id": "MCCcoiWd2uG1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load the pre-trained Whisper model for speech recognition\n",
        "Here we load the pre-trained Whisper for speech recognition.\n",
        "\n",
        "こちらでは音声認識に必要なWhisperを読み込みます。"
      ],
      "metadata": {
        "id": "qCD6CuNGyozZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n",
        "model_id = \"openai/whisper-large-v3\"\n",
        "\n",
        "w_model = AutoModelForSpeechSeq2Seq.from_pretrained(\n",
        "    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n",
        ")\n",
        "w_model.to(device)\n",
        "\n",
        "processor = AutoProcessor.from_pretrained(model_id)\n",
        "\n",
        "speech_recognizer = pipeline(\"automatic-speech-recognition\", model=w_model, tokenizer=processor.tokenizer, feature_extractor=processor.feature_extractor, torch_dtype=torch_dtype, device=device)"
      ],
      "metadata": {
        "id": "qC014e2Ty8ue"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load a pre-trained LLM\n",
        "We load a pre-trained LLM, that is Phi-3.5.\n",
        "\n",
        "学習済みLLMであるPhi-3.5を読み込みます。"
      ],
      "metadata": {
        "id": "eLn4_QUJ1vsJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = AutoModelForCausalLM.from_pretrained(\"microsoft/Phi-3.5-mini-instruct\", device_map=\"cuda\", torch_dtype=\"auto\", trust_remote_code=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3.5-mini-instruct\")\n",
        "generator = pipeline(\"text-generation\", model=model, tokenizer=tokenizer)"
      ],
      "metadata": {
        "id": "XjimTr2b1-EL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Record your voice\n",
        "You can record your voice to say to the dialogue system.<br>\n",
        "To use this function, you need to permit authority on the dialogue window that demand to use microphone.<br>\n",
        "Push `録音開始` button to start recording and push `録音停止` button to finish recording.<br>\n",
        "During recording, the message <font color=\"red\">録音中</font> will be displayed.<br>\n",
        "You can push `録音開始` and `録音停止` button whenever you want to record new voice.<br>\n",
        "To be more detailed, it uses JavaScript to record your voice internally.\n",
        "\n",
        "対話システムに渡すための音声を録音することができます。<br>\n",
        "この機能を使用されるにあたっては、マイクの利用を許可する旨のダイアログにおいて、権限を許可する必要があります。<br>\n",
        "`録音開始`ボタンを押すと録音を開始でき、`録音停止`ボタンを押すと録音を終了できます。<br>\n",
        "録音中は、<font color=\"red\">録音中</font> という文字が表示されます。<br>\n",
        "新しい音声を録音したいとき、`録音開始`ボタンおよび`録音停止`ボタンは何回でも押せます。<br>\n",
        "より詳しくお伝えすると、ここでは内部的には音声を録音するためにJavaScriptを使用しています。"
      ],
      "metadata": {
        "id": "PMHt0wCZ0m5S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import Javascript, display\n",
        "from google.colab import output\n",
        "import base64\n",
        "\n",
        "# JavaScript: 録音制御および「録音中」表示の更新\n",
        "RECORD_JS = \"\"\"\n",
        "const sleep = time => new Promise(resolve => setTimeout(resolve, time));\n",
        "var mediaRecorder = null;\n",
        "var audioChunks = [];\n",
        "\n",
        "function startRecording() {\n",
        "  navigator.mediaDevices.getUserMedia({ audio: true })\n",
        "    .then(stream => {\n",
        "      mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });\n",
        "      mediaRecorder.start();\n",
        "      audioChunks = [];\n",
        "\n",
        "      mediaRecorder.addEventListener('dataavailable', event => {\n",
        "        audioChunks.push(event.data);\n",
        "      });\n",
        "\n",
        "      mediaRecorder.addEventListener('stop', () => {\n",
        "        const audioBlob = new Blob(audioChunks, { type: 'audio/wav' });\n",
        "        const reader = new FileReader();\n",
        "        reader.readAsDataURL(audioBlob);  // Base64に変換\n",
        "\n",
        "        reader.onloadend = () => {\n",
        "          const base64String = reader.result.split(',')[1];  // データ部分のみ抽出\n",
        "          google.colab.kernel.invokeFunction('notebook.save_audio', [base64String], {});\n",
        "        };\n",
        "      });\n",
        "\n",
        "      // 録音中のステータスを表示\n",
        "      document.getElementById('status').textContent = '録音中...';\n",
        "    });\n",
        "}\n",
        "\n",
        "function stopRecording() {\n",
        "  if (mediaRecorder) {\n",
        "    mediaRecorder.stop();\n",
        "    // 録音中のステータスを消去\n",
        "    document.getElementById('status').textContent = '';\n",
        "  }\n",
        "}\n",
        "\"\"\"\n",
        "\n",
        "# JavaScriptコードをColabにロードする\n",
        "display(Javascript(RECORD_JS))\n",
        "\n",
        "# Python: 録音結果を処理し保存する\n",
        "def save_audio(base64_string):\n",
        "    # Reference: https://qiita.com/hiroyuki_kawahara/items/6a221f33d9c5078c3036\n",
        "    audio_bytes = base64.b64decode(base64_string)  # Base64をデコード\n",
        "\n",
        "    with open(audio_file, 'wb') as f:\n",
        "        f.write(audio_bytes)\n",
        "\n",
        "    print(f\"録音した音声を '{audio_file}' に保存しました。\")\n",
        "\n",
        "# JavaScriptからのコールバックを登録\n",
        "output.register_callback('notebook.save_audio', save_audio)\n",
        "\n",
        "# 録音用のUIを表示する関数\n",
        "def display_audio_recorder():\n",
        "    display(Javascript(\"\"\"\n",
        "        const container = document.createElement('div');\n",
        "        container.style.marginBottom = '10px';\n",
        "\n",
        "        const startButton = document.createElement('button');\n",
        "        startButton.textContent = '録音開始';\n",
        "        startButton.style.marginRight = '10px';\n",
        "        startButton.onclick = startRecording;\n",
        "\n",
        "        const stopButton = document.createElement('button');\n",
        "        stopButton.textContent = '録音停止';\n",
        "        stopButton.onclick = stopRecording;\n",
        "\n",
        "        const status = document.createElement('span');\n",
        "        status.id = 'status';\n",
        "        status.style.marginLeft = '10px';\n",
        "        status.style.color = 'red';\n",
        "\n",
        "        container.appendChild(startButton);\n",
        "        container.appendChild(stopButton);\n",
        "        container.appendChild(status);\n",
        "\n",
        "        document.body.appendChild(container);\n",
        "    \"\"\"))\n",
        "\n",
        "display_audio_recorder()\n"
      ],
      "metadata": {
        "id": "NO-rT6ph81g1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Perform speech recognition on a sample audio\n",
        "\n",
        "Here this cell recognize the voice as a text format.<br>\n",
        "The result of this recognition is shown on the following of the cell.\n",
        "\n",
        "ここで、このセルは音声をテキスト形式で認識しています。<br>\n",
        "認識結果はこのセルの下に表示されます。"
      ],
      "metadata": {
        "id": "PY7JZD3009Rh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = speech_recognizer(audio_file, generate_kwargs={\"language\": language})[\"text\"]\n",
        "print(f\"Recognized Speech: {text}\")"
      ],
      "metadata": {
        "id": "NhMrwJOS1U4M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set what the system looks like\n",
        "Here please set how the system is.<br>\n",
        "For example, you can arrange the system like some experts or developer and so on.<br>\n",
        "Here you should set it in English.\n",
        "\n",
        "ここでは、システムがどういった性格を持つかを指定してください。<br>\n",
        "例えば、そのシステムがエキスパートであったり開発者であったりと指定できます。<br>\n",
        "ここは、英語で指定してください。"
      ],
      "metadata": {
        "id": "aauZMbu35ATD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "character = \"expert for writing Python\""
      ],
      "metadata": {
        "id": "vYSo7qHn5Aqz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Use LLM to generate a response based on the recognized text\n",
        "Here we use LLM to generate a response based on the recognized text.<br>\n",
        "Did you satisfy with the reply with LLM?\n",
        "\n",
        "こちらで認識されたテキストについてLLMが返答を生成します。<br>\n",
        "LLMが生成した結果には満足していただけましたか？"
      ],
      "metadata": {
        "id": "DEIqv79k1nvR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "messages = [\n",
        "    {\"role\": \"system\", \"content\": \"You are {} {}.\".format(\"an\" if character[0] in ['a', 'i', 'u', 'e', 'o'] else \"a\", character)},\n",
        "    {\"role\": \"user\", \"content\": text},\n",
        "]\n",
        "response = generator(messages, max_new_tokens=500, return_full_text=False, temperature=0.0, do_sample=False)\n",
        "print(f\"Generated Response: {response[0]['generated_text']}\")"
      ],
      "metadata": {
        "id": "PYEH2xxc2UUY"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}