4-4-8-Flatbuffers.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S9eLyYdy9t4I"
      },
      "source": [
        "# TensorFlow Lite Flatbuffer Manipulation Example\n",
        "\n",
        "It's possible to read, modify, and write TensorFlow Lite model files from Python. This notebook shows you how.\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tinyMLx/colabs/blob/master/4-4-8-Flatbuffers.ipynb\n",
        "\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tinyMLx/colabs/blob/master/4-4-8-Flatbuffers.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LN0slo1N-dMT"
      },
      "source": [
        "## Software installation\n",
        "\n",
        "The goal is to build a set of Python classes that represent the types stored inside the TFL Flatbuffer files. To do this we need several dependencies:\n",
        " - The 'flatc' compiler, that converts the model format stored in a text schema to Python accessor classes.\n",
        " - The text schema itself, describing the model format.\n",
        " - The Flatbuffer Python library that the accessor classes rely on.\n",
        "\n",
        "The 'flatc' compiler isn't available as a binary download, so we need to build it from source. The version has to match the Flatbuffer Python library on the system, or the resulting generated code won't work, since it will be trying to use a different API. The FB Python library is at version 1.12.0, so we make sure we download the tagged snapshot of the source at that same version from GitHub."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gvc8Gv806odl"
      },
      "source": [
        "### Install Flatbuffer Python Library\n",
        "\n",
        "We should already have version 1.12.0 by default on this Colab, but use pip to ensure it's installed, and then import it so it's available from Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "v9eegi_vtxW4",
        "outputId": "b1b0deff-6e78-44cd-99f3-855632f4962c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: flatbuffers==23.5.26 in /usr/local/lib/python3.10/dist-packages (23.5.26)\n"
          ]
        }
      ],
      "source": [
        "!pip install flatbuffers==23.5.26\n",
        "import flatbuffers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "njMMlz3L69Re"
      },
      "source": [
        "### Build the 'flatc' Compiler\n",
        "\n",
        "We need 'flatc' to generate the Python accessor classes we'll use to read and write the serialized files, and since it's not easily available as a binary, we grab the source code from the right version and build it directly. **This will take a few minutes.**\n",
        "\n",
        "Once the 'flatc' binary has been built, copy it to the `/usr/local/bin` folder so that it's easily accessible as a command."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "HiM0ZsxO6NuX",
        "outputId": "f34b23e7-69a7-4a1f-cec7-b533e31879e6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100 3134k    0 3134k    0     0  2313k      0 --:--:--  0:00:01 --:--:-- 4496k\n",
            "/content/flatbuffers\n",
            "\u001b[33mCMake Warning:\n",
            "  No source or binary directory provided.  Both will be assumed to be the\n",
            "  same as the current working directory, but note that this warning will\n",
            "  become a fatal error in future CMake releases.\n",
            "\n",
            "\u001b[0m\n",
            "-- Proceeding with version: 23.5.26.0\n",
            "-- The CXX compiler identification is GNU 11.4.0\n",
            "-- Detecting CXX compiler ABI info\n",
            "-- Detecting CXX compiler ABI info - done\n",
            "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n",
            "-- Detecting CXX compile features\n",
            "-- Detecting CXX compile features - done\n",
            "-- Looking for strtof_l\n",
            "-- Looking for strtof_l - found\n",
            "-- Looking for strtoull_l\n",
            "-- Looking for strtoull_l - found\n",
            "-- Looking for realpath\n",
            "-- Looking for realpath - found\n",
            "-- CMAKE_CXX_FLAGS: \n",
            "-- Configuring done (0.8s)\n",
            "-- Generating done (0.0s)\n",
            "-- Build files have been written to: /content/flatbuffers\n",
            "[  2%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/reflection.cpp.o\u001b[0m\n",
            "[  1%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[  3%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[  5%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/util.cpp.o\u001b[0m\n",
            "[  6%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[  6%] \u001b[32mBuilding CXX object CMakeFiles/flatbuffers.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[  7%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/util.cpp.o\u001b[0m\n",
            "[  8%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 10%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_binary.cpp.o\u001b[0m\n",
            "[ 11%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_cpp.cpp.o\u001b[0m\n",
            "[ 12%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_csharp.cpp.o\u001b[0m\n",
            "[ 13%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_dart.cpp.o\u001b[0m\n",
            "[ 14%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_kotlin.cpp.o\u001b[0m\n",
            "[ 15%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_go.cpp.o\u001b[0m\n",
            "[ 16%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_java.cpp.o\u001b[0m\n",
            "[ 17%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_ts.cpp.o\u001b[0m\n",
            "[ 18%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_php.cpp.o\u001b[0m\n",
            "[ 20%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_python.cpp.o\u001b[0m\n",
            "[ 21%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_lobster.cpp.o\u001b[0m\n",
            "[ 22%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_rust.cpp.o\u001b[0m\n",
            "[ 23%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_fbs.cpp.o\u001b[0m\n",
            "[ 24%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_grpc.cpp.o\u001b[0m\n",
            "[ 25%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_json_schema.cpp.o\u001b[0m\n",
            "[ 26%] \u001b[32m\u001b[1mLinking CXX static library libflatbuffers.a\u001b[0m\n",
            "[ 26%] Built target flatbuffers\n",
            "[ 27%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/idl_gen_swift.cpp.o\u001b[0m\n",
            "[ 28%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/file_name_saving_file_manager.cpp.o\u001b[0m\n",
            "[ 30%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/file_binary_writer.cpp.o\u001b[0m\n",
            "[ 31%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/file_writer.cpp.o\u001b[0m\n",
            "[ 32%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/flatc.cpp.o\u001b[0m\n",
            "[ 33%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/flatc_main.cpp.o\u001b[0m\n",
            "[ 34%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/binary_annotator.cpp.o\u001b[0m\n",
            "[ 35%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/annotated_binary_text_gen.cpp.o\u001b[0m\n",
            "[ 36%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/bfbs_gen_lua.cpp.o\u001b[0m\n",
            "[ 37%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/bfbs_gen_nim.cpp.o\u001b[0m\n",
            "[ 38%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/src/code_generators.cpp.o\u001b[0m\n",
            "[ 40%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/cpp_generator.cc.o\u001b[0m\n",
            "[ 41%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/go_generator.cc.o\u001b[0m\n",
            "[ 42%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/java_generator.cc.o\u001b[0m\n",
            "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/python_generator.cc.o\u001b[0m\n",
            "[ 44%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/swift_generator.cc.o\u001b[0m\n",
            "[ 45%] \u001b[32mBuilding CXX object CMakeFiles/flatc.dir/grpc/src/compiler/ts_generator.cc.o\u001b[0m\n",
            "[ 46%] \u001b[32m\u001b[1mLinking CXX executable flatc\u001b[0m\n",
            "[ 46%] Built target flatc\n",
            "[ 47%] \u001b[34m\u001b[1mflatc generation: `tests/union_underlying_type_test.fbs` -> `tests/union_underlying_type_test_generated.h`\u001b[0m\n",
            "[ 48%] \u001b[34m\u001b[1mflatc generation: `tests/64bit/evolution/v1.fbs` -> `tests/64bit/evolution/v1_generated.h`\u001b[0m\n",
            "[ 50%] \u001b[34m\u001b[1mflatc generation: `tests/64bit/evolution/v2.fbs` -> `tests/64bit/evolution/v2_generated.h`\u001b[0m\n",
            "[ 51%] \u001b[34m\u001b[1mflatc generation: `samples/monster.fbs` -> `samples/monster_generated.h`\u001b[0m\n",
            "[ 52%] \u001b[34m\u001b[1mflatc generation: `tests/64bit/test_64bit.fbs` -> `tests/64bit/test_64bit_generated.h`\u001b[0m\n",
            "[ 53%] \u001b[34m\u001b[1mflatc generation: `tests/alignment_test.fbs` -> `tests/alignment_test_generated.h`\u001b[0m\n",
            "[ 53%] Built target flatsample\n",
            "[ 54%] \u001b[34m\u001b[1mflatc generation: `tests/arrays_test.fbs` -> `tests/arrays_test_generated.h`\u001b[0m\n",
            "[ 55%] \u001b[34m\u001b[1mflatc generation: `tests/key_field/key_field_sample.fbs` -> `tests/key_field/key_field_sample_generated.h`\u001b[0m\n",
            "[ 56%] \u001b[34m\u001b[1mflatc generation: `tests/native_inline_table_test.fbs` -> `tests/native_inline_table_test_generated.h`\u001b[0m\n",
            "[ 58%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 57%] \u001b[34m\u001b[1mflatc generation: `tests/native_type_test.fbs` -> `tests/native_type_test_generated.h`\u001b[0m\n",
            "[ 60%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 61%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebinary.dir/samples/sample_binary.cpp.o\u001b[0m\n",
            "[ 62%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 63%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 64%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 65%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 66%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_parser.cpp.o\u001b[0m\n",
            "[ 67%] \u001b[32m\u001b[1mLinking CXX executable flatsamplebinary\u001b[0m\n",
            "[ 67%] Built target flatsamplebinary\n",
            "[ 68%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_gen_text.cpp.o\u001b[0m\n",
            "[ 70%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/reflection.cpp.o\u001b[0m\n",
            "[ 71%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/src/util.cpp.o\u001b[0m\n",
            "[ 72%] \u001b[32mBuilding CXX object CMakeFiles/flatsampletext.dir/samples/sample_text.cpp.o\u001b[0m\n",
            "[ 73%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/src/util.cpp.o\u001b[0m\n",
            "[ 74%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/util.cpp.o\u001b[0m\n",
            "[ 75%] \u001b[32mBuilding CXX object CMakeFiles/flatsamplebfbs.dir/samples/sample_bfbs.cpp.o\u001b[0m\n",
            "[ 76%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/idl_gen_fbs.cpp.o\u001b[0m\n",
            "[ 77%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/evolution_test.cpp.o\u001b[0m\n",
            "[ 78%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/flexbuffers_test.cpp.o\u001b[0m\n",
            "[ 80%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/fuzz_test.cpp.o\u001b[0m\n",
            "[ 81%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/json_test.cpp.o\u001b[0m\n",
            "[ 82%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/key_field_test.cpp.o\u001b[0m\n",
            "[ 83%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/monster_test.cpp.o\u001b[0m\n",
            "[ 84%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/optional_scalars_test.cpp.o\u001b[0m\n",
            "[ 85%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/parser_test.cpp.o\u001b[0m\n",
            "[ 86%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/proto_test.cpp.o\u001b[0m\n",
            "[ 87%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/reflection_test.cpp.o\u001b[0m\n",
            "[ 88%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test.cpp.o\u001b[0m\n",
            "[ 90%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test_assert.cpp.o\u001b[0m\n",
            "[ 91%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/test_builder.cpp.o\u001b[0m\n",
            "[ 92%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/util_test.cpp.o\u001b[0m\n",
            "[ 93%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/native_type_test_impl.cpp.o\u001b[0m\n",
            "[ 94%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/alignment_test.cpp.o\u001b[0m\n",
            "[ 95%] \u001b[32m\u001b[1mLinking CXX executable flatsampletext\u001b[0m\n",
            "[ 95%] Built target flatsampletext\n",
            "[ 96%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/tests/64bit/offset64_test.cpp.o\u001b[0m\n",
            "[ 97%] \u001b[32mBuilding CXX object CMakeFiles/flattests.dir/src/code_generators.cpp.o\u001b[0m\n",
            "[ 98%] \u001b[32m\u001b[1mLinking CXX executable flatsamplebfbs\u001b[0m\n",
            "[ 98%] Built target flatsamplebfbs\n",
            "[100%] \u001b[32m\u001b[1mLinking CXX executable flattests\u001b[0m\n",
            "[100%] Built target flattests\n"
          ]
        }
      ],
      "source": [
        "# Build and install the Flatbuffer compiler.\n",
        "%cd /content/\n",
        "!rm -rf flatbuffers*\n",
        "!curl -L \"https://github.com/google/flatbuffers/archive/v23.5.26.zip\" -o flatbuffers.zip\n",
        "!unzip -q flatbuffers.zip\n",
        "!mv flatbuffers-23.5.26 flatbuffers\n",
        "%cd flatbuffers\n",
        "!cmake -G \"Unix Makefiles\" -DCMAKE_BUILD_TYPE=Release\n",
        "!make -j 8\n",
        "!cp flatc /usr/local/bin/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D5vlh6BCM9i0"
      },
      "source": [
        "### Fetch the Schema\n",
        "\n",
        "The schema is [a text file that describes the layout of the model file format](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema_v3.fbs), and it's part of the TensorFlow source code, so we need to fetch the latest version from GitHub."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "4boI3wM00PnS",
        "outputId": "d4606a83-691b-460b-8251-2a08af5234ea",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n",
            "Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.\n",
            "--2023-09-25 19:19:08--  https://github.com/tensorflow/tensorflow/archive/v2.4.1.zip\n",
            "Resolving github.com (github.com)... 140.82.114.3\n",
            "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://codeload.github.com/tensorflow/tensorflow/zip/refs/tags/v2.4.1 [following]\n",
            "--2023-09-25 19:19:08--  https://codeload.github.com/tensorflow/tensorflow/zip/refs/tags/v2.4.1\n",
            "Resolving codeload.github.com (codeload.github.com)... 140.82.114.9\n",
            "Connecting to codeload.github.com (codeload.github.com)|140.82.114.9|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [application/zip]\n",
            "Saving to: ‘v2.4.1.zip’\n",
            "\n",
            "v2.4.1.zip              [   <=>              ]  66.13M  12.2MB/s    in 6.3s    \n",
            "\n",
            "2023-09-25 19:19:15 (10.5 MB/s) - ‘v2.4.1.zip’ saved [69346072]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "%cd /content/\n",
        "!rm -rf tensorflow\n",
        "%tensorflow_version 2.x\n",
        "!wget https://github.com/tensorflow/tensorflow/archive/v2.4.1.zip\n",
        "!unzip v2.4.1.zip &> 0\n",
        "!mv tensorflow-2.4.1/ tensorflow/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GS1eAfwRNfvG"
      },
      "source": [
        "### Generate the Python Accessor Classes\n",
        "\n",
        "The 'flatc' compiler takes the information from the text schema, and creates Python code to read and write the data held inside the serialized Flatbuffer file. The Python classes are written into the `tflite` folder, with names like `Model.py`, and define classes like `ModelT` that contain members that you can use to read or write the contents of the data structures defined by the schema."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Xl0_MIlMM6Es"
      },
      "outputs": [],
      "source": [
        "!flatc --python --gen-object-api tensorflow/tensorflow/lite/schema/schema_v3.fbs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8p_5vLNQ_sFF"
      },
      "source": [
        "### Model Loading and Saving\n",
        "\n",
        "These are some simple wrapper functions that demonstrate how to load data from a file, turn it into a `ModelT` Python object that can be modified, and then save it out to a new file.\n",
        "\n",
        "We have to do a little bit of hackery with the Python search paths so that the class files we generated from the schema can be imported."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "acpsc72lvKRW"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "# This hackery allows us to import the Python files we've just generated.\n",
        "sys.path.append(\"/content/tflite/\")\n",
        "import Model\n",
        "\n",
        "def load_model_from_file(model_filename):\n",
        "  with open(model_filename, \"rb\") as file:\n",
        "    buffer_data = file.read()\n",
        "  model_obj = Model.Model.GetRootAsModel(buffer_data, 0)\n",
        "  model = Model.ModelT.InitFromObj(model_obj)\n",
        "  return model\n",
        "\n",
        "def save_model_to_file(model, model_filename):\n",
        "  builder = flatbuffers.Builder(1024)\n",
        "  model_offset = model.Pack(builder)\n",
        "  builder.Finish(model_offset, file_identifier=b'TFL3')\n",
        "  model_data = builder.Output()\n",
        "  with open(model_filename, 'wb') as out_file:\n",
        "    out_file.write(model_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtNYC984AILB"
      },
      "source": [
        "## Download an Example Model\n",
        "\n",
        "This pulls down a trained model from the speech commands tutorial that we can use to test the Flatbuffer loading code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "56u94mxKsCEM",
        "outputId": "a4dde3f7-ef4a-4f7d-a5dc-c538524fd6e6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 78855  100 78855    0     0   251k      0 --:--:-- --:--:-- --:--:--  252k\n",
            "Archive:  speech_commands_model_2020_04_27.zip\n",
            "  inflating: speech_commands_model/speech_commands_model_float.tflite  \n",
            "  inflating: speech_commands_model/speech_commands_model_int8.tflite  \n"
          ]
        }
      ],
      "source": [
        "!curl -O 'https://storage.googleapis.com/download.tensorflow.org/models/tflite/micro/speech_commands_model_2020_04_27.zip'\n",
        "!unzip speech_commands_model_2020_04_27.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5SJTcrRBy1X"
      },
      "source": [
        "## Load, Modify, and Save a Model\n",
        "\n",
        "The code below loads the float model, applies a small change to the float weights, and saves it out again. In this case we're just changing the contents of the weights, but any of the other properties of the model can be added, removed, or modified, including tensors, ops, and metadata.\n",
        "\n",
        "I've found the easiest way to understand the syntax used and data available is to look at the generated classes in `tflite/` and [the text schema that describes the format](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema_v3.fbs)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "zex9zZo01lM4"
      },
      "outputs": [],
      "source": [
        "# Use numpy to make the manipulation of the weight arrays easier.\n",
        "import numpy as np\n",
        "\n",
        "# Load the float model we downloaded as a ModelT object.\n",
        "model = load_model_from_file('/content/speech_commands_model/speech_commands_model_float.tflite')\n",
        "\n",
        "# Loop through all the weights held in the model.\n",
        "for buffer in model.buffers:\n",
        "  # Skip missing or small weight arrays.\n",
        "  if buffer.data is not None and len(buffer.data) > 1024:\n",
        "    # Pull the weight array from the model, and cast it to 32-bit floats since\n",
        "    # we know that's the type for all the weights in this model. In a real\n",
        "    # application we'd need to check the data type from the tensor information\n",
        "    # stored in the model.subgraphs\n",
        "    original_weights = np.frombuffer(buffer.data, dtype=np.float32)\n",
        "\n",
        "    # This is the line where the weights are altered.\n",
        "    # Try replacing it with your own version, for example:\n",
        "    # munged_weights = np.add(original_weights, 0.002)\n",
        "    munged_weights = np.round(original_weights * (1/0.02)) * 0.02\n",
        "\n",
        "    # Write the altered data back into the model.\n",
        "    buffer.data = munged_weights.tobytes()\n",
        "\n",
        "# Save the ModelT object as a TFL Flatbuffer file.\n",
        "save_model_to_file(model, '/content/speech_commands_model/speech_commands_model_modified.tflite')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8EwrywLX9gs"
      },
      "source": [
        "## Evaluating the impact of your changes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_EAgJJNmZL8T"
      },
      "source": [
        "### Setup KWS infrastructure\n",
        "\n",
        "To evaluate the impact of your changes on the accuracy of the speech model we need to load a test data set and some utility classes to read the files and convert them into the right input form for the network.\n",
        "\n",
        "**The full dataset is several gigabytes in size, so it may take a few minutes to download.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nlrzIA__X9v3",
        "outputId": "edc861d7-d8f6-45a8-b17b-ffde4bd9c06f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ">> Downloading speech_commands_v0.02.tar.gz 100.0%\n"
          ]
        }
      ],
      "source": [
        "sys.path.append(\"/content/tensorflow/tensorflow/examples/speech_commands/\")\n",
        "import input_data\n",
        "import models\n",
        "\n",
        "# A comma-delimited list of the words you want to train for.\n",
        "# The options are: yes,no,up,down,left,right,on,off,stop,go\n",
        "# All the other words will be used to train an \"unknown\" label and silent\n",
        "# audio data with no spoken words will be used to train a \"silence\" label.\n",
        "WANTED_WORDS = \"yes,no\"\n",
        "\n",
        "# The number of steps and learning rates can be specified as comma-separated\n",
        "# lists to define the rate at each stage. For example,\n",
        "# TRAINING_STEPS=12000,3000 and LEARNING_RATE=0.001,0.0001\n",
        "# will run 12,000 training loops in total, with a rate of 0.001 for the first\n",
        "# 8,000, and 0.0001 for the final 3,000.\n",
        "TRAINING_STEPS = \"12000,3000\"\n",
        "LEARNING_RATE = \"0.001,0.0001\"\n",
        "\n",
        "# Calculate the total number of steps, which is used to identify the checkpoint\n",
        "# file name.\n",
        "TOTAL_STEPS = str(sum(map(lambda string: int(string), TRAINING_STEPS.split(\",\"))))\n",
        "\n",
        "# Calculate the percentage of 'silence' and 'unknown' training samples required\n",
        "# to ensure that we have equal number of samples for each label.\n",
        "number_of_labels = WANTED_WORDS.count(',') + 1\n",
        "number_of_total_labels = number_of_labels + 2 # for 'silence' and 'unknown' label\n",
        "equal_percentage_of_training_samples = int(100.0/(number_of_total_labels))\n",
        "SILENT_PERCENTAGE = equal_percentage_of_training_samples\n",
        "UNKNOWN_PERCENTAGE = equal_percentage_of_training_samples\n",
        "\n",
        "# Constants which are shared during training and inference\n",
        "PREPROCESS = 'micro'\n",
        "WINDOW_STRIDE =20\n",
        "MODEL_ARCHITECTURE = 'tiny_conv' # Other options include: single_fc, conv,\n",
        "                      # low_latency_conv, low_latency_svdf, tiny_embedding_conv\n",
        "\n",
        "# Constants used during training only\n",
        "VERBOSITY = 'WARN'\n",
        "EVAL_STEP_INTERVAL = '1000'\n",
        "SAVE_STEP_INTERVAL = '1000'\n",
        "\n",
        "# Constants for training directories and filepaths\n",
        "DATASET_DIR =  'dataset/'\n",
        "LOGS_DIR = 'logs/'\n",
        "TRAIN_DIR = 'train/' # for training checkpoints and other files.\n",
        "\n",
        "SAMPLE_RATE = 16000\n",
        "CLIP_DURATION_MS = 1000\n",
        "WINDOW_SIZE_MS = 30.0\n",
        "FEATURE_BIN_COUNT = 40\n",
        "BACKGROUND_FREQUENCY = 0.8\n",
        "BACKGROUND_VOLUME_RANGE = 0.1\n",
        "TIME_SHIFT_MS = 100.0\n",
        "\n",
        "DATA_URL = 'https://storage.googleapis.com/download.tensorflow.org/data/speech_commands_v0.02.tar.gz'\n",
        "VALIDATION_PERCENTAGE = 10\n",
        "TESTING_PERCENTAGE = 10\n",
        "\n",
        "model_settings = models.prepare_model_settings(\n",
        "    len(input_data.prepare_words_list(WANTED_WORDS.split(','))),\n",
        "    SAMPLE_RATE, CLIP_DURATION_MS, WINDOW_SIZE_MS,\n",
        "    WINDOW_STRIDE, FEATURE_BIN_COUNT, PREPROCESS)\n",
        "audio_processor = input_data.AudioProcessor(\n",
        "    DATA_URL, DATASET_DIR,\n",
        "    SILENT_PERCENTAGE, UNKNOWN_PERCENTAGE,\n",
        "    WANTED_WORDS.split(','), VALIDATION_PERCENTAGE,\n",
        "    TESTING_PERCENTAGE, model_settings, LOGS_DIR)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "bMsyppDEYaYP"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "# define our helper function to test the model accuracy\n",
        "def test_model_accuracy(model_filename):\n",
        "  with tf.compat.v1.Session() as sess:\n",
        "    test_data, test_labels = audio_processor.get_data(\n",
        "        -1, 0, model_settings, 0, 0,\n",
        "        0, 'testing', sess)\n",
        "\n",
        "  interpreter = tf.lite.Interpreter(model_filename)\n",
        "  interpreter.allocate_tensors()\n",
        "\n",
        "  input_index = interpreter.get_input_details()[0][\"index\"]\n",
        "\n",
        "  output_index = interpreter.get_output_details()[0][\"index\"]\n",
        "  model_output = interpreter.tensor(output_index)\n",
        "\n",
        "  correct_predictions = 0\n",
        "  for i in range(len(test_data)):\n",
        "    current_input = test_data[i]\n",
        "    current_label = test_labels[i]\n",
        "    flattened_input = np.array(current_input.flatten(), dtype=np.float32).reshape(1, 1960)\n",
        "    interpreter.set_tensor(input_index, flattened_input)\n",
        "    interpreter.invoke()\n",
        "    top_prediction = model_output()[0].argmax()\n",
        "    if top_prediction == current_label:\n",
        "      correct_predictions += 1\n",
        "\n",
        "  print('Accuracy is %f%% (N=%d)' % ((correct_predictions * 100) / len(test_data), len(test_data)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLSPYyKxYoNT"
      },
      "source": [
        "### Test your Models\n",
        "\n",
        "Finally we can test both the standard and your modified models!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PDkPbCi_ZYJL"
      },
      "source": [
        "**Float model evaluation**\n",
        "\n",
        "You should see ~91% accuracy with a 67KB model. Note: the exact accuracy may vary by a percent or two as it tests the model on a random sampling of the dataset and therefore sometimes gets particularly lucky or unlucky!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "toSX48DGYjqP",
        "outputId": "ae3aa3d5-3973-496b-d981-3c9fefe5fd25",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-rw-r--r-- 1 root root 67K Apr 27  2020 /content/speech_commands_model/speech_commands_model_float.tflite\n",
            "Accuracy is 92.556634% (N=1236)\n"
          ]
        }
      ],
      "source": [
        "!ls -lah /content/speech_commands_model/speech_commands_model_float.tflite\n",
        "test_model_accuracy('/content/speech_commands_model/speech_commands_model_float.tflite')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RF3T18DWZe7J"
      },
      "source": [
        "**Modified model evaluation**\n",
        "\n",
        "Test the impact of your changes!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "UIfNzw7kZqga",
        "outputId": "cd63530f-d09b-4f06-a373-7323e8d29778",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-rw-r--r-- 1 root root 67K Sep 25 19:20 /content/speech_commands_model/speech_commands_model_modified.tflite\n",
            "Accuracy is 90.695793% (N=1236)\n"
          ]
        }
      ],
      "source": [
        "!ls -lah /content/speech_commands_model/speech_commands_model_modified.tflite\n",
        "test_model_accuracy('/content/speech_commands_model/speech_commands_model_modified.tflite')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vzw4-n8HAgtT"
      },
      "source": [
        "If you'd like to try more modified models just scroll back up to the **Load, Modify, and Save a Model** section and re-run the modified model generation step and then skip back down to the above to re-test!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "4-4-8-Flatbuffers.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}