vision/glm-v_for_video_understanding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing and narrating a video with ZhipuAI GLM's visual capabilities\n",
    "\n",
    "**This tutorial is available in English and is attached below the Chinese explanation**\n",
    "\n",
    "此代码演示了如何通过视频使用 GLM 的视觉功能。 GLM-4 不直接将视频作为输入，但我们可以使用视觉和新的 128K 上下文窗口来一次性描述整个视频的静态帧。\n",
    "\n",
    "**由于模型对视频理解的能力有待提高，在这个代码中的视频理解的细节程度无法达到较高水平。**\n",
    "\n",
    "This cookbook demonstrates how to use GLM's visual capabilities with a video. GLM-4 doesn't take videos as input directly, but we can use vision and the new 128K context window to describe the static frames of a whole video at once. \n",
    "\n",
    "**Since the model's ability to understand videos needs to be improved, the level of detail of video understanding in this code cannot reach a high level. **"
   ]
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "首先，设定好调用模型的API key\n",
    "\n",
    "First, set the API key for calling the model"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "import os\n",
    "from zhipuai import ZhipuAI\n",
    "os.environ[\"ZHIPUAI_API_KEY\"] = \"your zhipuai api key\"\n",
    "\n",
    "client = ZhipuAI()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-10-11T09:56:32.711637Z",
     "start_time": "2024-10-11T09:56:32.672578Z"
    }
   },
   "outputs": [],
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，将视频传入给模型，请注意，只有`GLM-4V-Plus` 模型支持对视频进行理解。接着，我们将要视频编码为 video_base64。 由大模型进行分析即可。\n",
    "\n",
    "Next, input the video into the model. Please note that only the `GLM-4V-Plus` model supports video understanding. After that, we need to encode the video into video_base64. The analysis can then be performed by the large model."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-11T09:56:44.364821Z",
     "start_time": "2024-10-11T09:56:32.713257Z"
    }
   },
   "source": [
    "import base64\n",
    "\n",
    "video_path = \"data/video_1.mp4\"\n",
    "with open(video_path, 'rb') as video_file:\n",
    "    video_base = base64.b64encode(video_file.read()).decode('utf-8')\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=\"glm-4v-plus\", \n",
    "    temperature=0.0,\n",
    "    top_p=0,\n",
    "    messages=[\n",
    "      {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "          {\n",
    "            \"type\": \"video_url\",\n",
    "            \"video_url\": {\n",
    "                \"url\" : video_base\n",
    "            }\n",
    "          },\n",
    "          {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": \"请仔细描述这个视频的环境，图中的小狗在干啥，以有趣的方式讲给我听\"\n",
    "          }\n",
    "        ]\n",
    "      }\n",
    "    ]\n",
    ")\n",
    "print(response.choices[0].message)\n"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompletionMessage(content='小狗在视频中正在玩耍,它正在跳进一个水坑里,溅起水花,看起来非常开心。它不停地跳进水坑里,又跑出来,然后再跳进去,好像永远都不会累一样。它的毛发湿透了,但似乎并不在意,只是不停地玩耍。有时候它会停下来,甩甩身上的水,然后再继续玩耍。看起来非常享受这个时刻,好像在享受水的清凉和自由的感觉。', role='assistant', tool_calls=None)\n"
     ]
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "通常来说，视频理解的响应时间较长，需要等待数十秒。\n",
    "\n",
    "Typically, video understanding has a longer response time, often requiring several tens of seconds to process."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}